Thursday, December 07, 2006

Agile Development -or- How to name a religion

Upon reading the normal set of useless arguments about Agile development (I'm not implying a side here actually, just saying the arguments just seem to go on and on), I've noticed how actually brilliant the naming of "Agile Development" really is.

It seems like it just makes sense right - I mean, "Our development cycle is nimble, quick, and Agile". But the real brilliance is *not* in the positive. Its really more brilliant in the negative.

What I mean is simply this. If someone asks you if you're doing Agile, and you aren't - you are now "not agile". I mean - your 4 person dev team could turn on a dime but follow none of the agile development principles, and YOU are the one termed "not agile" over some large team that has a standup meeting once a day.

This is really smart. If you're not with us, you're "not us". This can be extrapolated if you were deciding what to name a new religion. For example, a good candidate would be to name a new religion "Church of the Good Persons" where its members are indeed "good persons".

Then of course, if you are not in that religion you are implicitly "not a good person".

"Fred, I'd like you to meet Bob. Bob isn't a good person, but I'm trying to convince him to become a good person. Isn't that right Bob?"
"I, um, er...."

Fariggin awesome.

You're either with us or you're against us. And your development team is indeed either agile or ya know, its not.

Wednesday, December 06, 2006

The Architecture of Mailinator

Almost 3.5 years ago I started the Mailinator(tm) service. I got the bulk of the idea from my drunk roommate at the time and the first incarnation took me all of about 3 days to code up. In some senses it was a crazy idea. I imagine many people came up with a similar idea before - make a web-based email service that allowed any incoming email to create an inbox. No sign-up. No personal information. Send email first, check email later.



This became ridiculously handy for things like signing up for websites that send you one confirmation
email, then save or sell or spam your email address forever. And of course, it *is* very handy for users. But think about it from mailinator's side. Its basically signing up to receive spam for that address forever. That's a tall order and one that seems to have the possibility of a terrible demise. Someday, enough email could come in that will simply smush Mailinator. But, as of this writing, that day isn't today.

I have in that 3.5 years received hundreds of "thank you" emails, a pile of "it doesn't work" emails, a radio interview, articles in the Washington Post, New York Times, and Delta Skymiles magazine, 1 call from both
Scotland Yard and the LAPD, and a total of 4 subpoenas (1 of those being a Federal Grand Jury subpoena issued by the FBI).

At this point, Mailinator averages approximately 2.5million emails per day. I have seen hourly spikes that would result in about 5million in a day. In addition, the system also services several thousand web users and several thousand RSS users per day.


In the world of email services, this probably isn't all that much. The most interesting part to me is that the complete set of hardware that mailinator uses is one little server. Just one. A very modest machine with an AMD 2Ghz Athlon processor, 1G of ram (although it really doesn't need that much), and a boring (IDE , low-performance) 80G hard drive. And honestly, its really not very busy at all. I've read the blogs of some copycat services of Mailinator where their owners were upgrading their servers to some big iron. This was really the impetus for me writing down this document - to share a different point of view.

Mailinator easily handling a few million emails a day wasn't always the case. The initial mailinator system was quite busy. And in fact, got overwhelmed about a year ago when email traffic started topping 800,000 a day (that's my recollection anyway). In an effort to squeeze life out of the
server and as an exercise in putting together some principles I always championed about server development, I rewrote the system from scratch. I have no idea what the current limit of the existing system is, but at 2.5million a day, its not even breaking a sweat.

If you don't know what Mailinator is, take a small tour through the (rather funny) FAQ.

Lossy lossie lossee

There is a very important point to note about the Mailinator service. And that is, that it is indeed - free. Although it might not seem like it, it has an immense impact on the design (as you'll see). This allowed me to favor performance across the board of the design. This fact influenced decisions from how I dealt with detecting spam all the way down to how I synchronized some code blocks. No kidding.

The basic tenet is that I do not have to provide perfect service. In order to do that, my hardware requirements would be much higher. Now that would all be fine and dandy if people were paying for the service. I could then provide support and guarantees. But given its free I instead went for, in order, these two design decisions:

1) Design a system that values survival above all else even users (as of course, if its down, users aren't really getting much out of it)
2) Provide 99.99% uptime and accuracy for users.

If you wonder what I mean about "survival" in the first line, it basically means that Mailinator is attacked on literally a daily basis. I wanted to make a system that could survive the large majority of those attacks. Note - I'm not interested in it surviving all of them. Because again, if some zombie network decided to Denial-of-service me - I really have no chance of thwarting it without some serious hardware. The good news is that if someone goes to all the trouble of smashing Mailinator (again referencing the fact we're lossy), I really don't lose much sleep over it. It sucks for my users - but there really isn't anything I can do anyway. I'm not trying to be cavalier about this - I went to great lengths to handle attacks, I'm just saying its a cold reality that I simply cannot stop them all. Thus I accept them as part of the game.

The platform

The original Mailinator used a relatively standard
unix stack of applications including a Java based web application running in Tomcat. Mailinator is and was of course, always just a hobby. I had a day job (or 2) so months of development just was never an option. I chose Java for no other reason than I knew Java better than anything else. For email, it used sendmail with a special rule that directed any incoming email to mailinator.com to one single mailbox.

Sendmail --> disk --> Mailinator <-- Tomcat Servlet Engine

The Java based mailinator app then grabbed the emails using IMAP and/or POP (it changed over time) and deleted them. I should have used an mbox interface but I never got around to implementing that. The system then loaded all emails into memory and let them sit there. Mailinator only allowed about 20000 emails to reside in memory at once. So when a new one came, the oldest one got pushed out.

The FAQ advertises that emails stick around for "a couple of hours." And that was true, but exactly how long mattered on the rate of incoming emails. You'll also note an interesting side effect that since all emails lived in memory, if the server came down - all emails were lost! Talk about exploiting the fact that my service was free huh? This may seem dubious but the code was really quite stable and ran for weeks and months without downtime.

I thought about saving emails into a database of course but honestly, all this bought me was emails that stuck around longer. And, that in and of itself sort of went against my intent for mailinator. The ideas was, sign-up for something, goto Mailinator, click the link, and forget about it. If you want a
mailbox
where emails last a few days, thats fine, but there are many other alternatives out there - that's not what Mailinator is about. I forgot the database idea and now shoot for mails that last somewhere around 3-4 hours.

This all worked fabulously for awhile. It pretty much filled up all 1G of ram of the server. Finally when the incoming email rate started surpassing 800,000 a day, the system started to break down. I believe it was primarily the disk contention between unix mail apps and the Java app locking mailboxes. Regardless, there were many issues with that system that bugged me for a long time. The root of most of those problems really boiled down to one thing - the disk. The disk activity of sendmail, procmail, logging and whatever else was a silly bottleneck. And it needed to go.

More than a year ago now I did a full rewrite. Much of the anti-spam code that I'll describe later was already in this code-base but was improved and extended for the new system.

Synchronous vs. Asynchronous I/O

I've read a fair number of articles on the wonders of asynchronous I/O (java's NIO library). I don't doubt them but I decided against using it. Primarily, again, because I did a great deal of work in multithreaded environments and knew that area well. I figured if I had performance issues later, I could always switch over to NIO as a learning experience.

The biggest thing I knew I needed to do with Mailinator was to remove the unix application components. Mailinator needed to stop outsourcing its email receipt and do it itself. This basically meant I needed to write my own SMTP server. Or at least, a subset of one. Firstly, Mailinator has never had the ability to send email so I didn't need to code that part up. Second, I had really different needs for receiving email. I wanted to get it as fast as possible -or- refuse it as fast as possible.

SMTP has a rich dialog for errors but I chose to only support one error message. And that error is, appropriately enough - "User Unknown". That's a touch ironic since Mailinator accepts any user at all. Simply said, if you do anything that the Mailinator server doesn't like - you'll get a user unknown error. Even if you haven't sent it the username yet.

I looked at Apache James as a base which is a pure java SMTP server but it was way too comprehensive for my needs. I really just found some code examples and the SMTP specs and wrote things basically from scratch. From there, I was able to get an email, parse it, and put it right into memory. This bypassed the old system's step of writing it to disk all the way. From wire to user, mailinator mail never touches the disk. In fact, the Mailinator server's
disk is pretty darn idle all things considered.

Now to address persistence concerns right away - Mailinator doesn't run diskless, but it does run very asynchronously with regards to the disk. Emails are not written to disk EVER unless the system is coming down and is instructed to write them first (so it can reload them upon reboot). This little fact has been very handy when I've been subpoenaed. I simply do not have access to any emails that were sent to Mailinator in the past. If it is possible that I can get an email - so can you just by checking that inbox. If you can't get it then that means its long deleted from memory and nothing is going to get it back.

Mailinator also used to do logging (again, shut-off because of pesky subpoenas). But it did it very "batchy". It wrote several thousand logs lines to memory before doing one disk write. In effect we never want to have contention based on the incredibly slow disk.

Now if this all sounds a bit shaky, as in we might just lose an email now and then - you're right. But remember, our goal is 99.99% accuracy. Not 100%. That's an important distinction. The latest incarnation of Mailinator literally runs for months unattended. We do lose emails once in awhile - but its rare and usually involves a server crash. We accept the loss and by far most users never encounter it.

Emails

The system now is one unit. The web application, the email server, and all email storage run in one JVM.


The system uses under 300 threads. I can increase that number but haven't seen a need as of yet. When an email arrives (or attempts to arrive) it must pass a strong set of filters that are described below. If it gets past those filters it is then stored in memory - however, it is first compressed to save in-memory space. Over 99% of emails that arrive are never looked at, so we only ever decompress an email if someone actually "looks" at them.

Because of this, I am able to store many more emails than the original system's 20000. The current mailinator stores about 80000 emails and uses under 300M or ram. I probably should increase this number as plenty of ram is just sitting around. The average email lifespan is about 3-4 hours with this pool. The amount of incoming email has gone way up, so even by increasing this pool, we're largely staying steady as far as email lifespan. I could probably kick that up to 200,000 or so and increase the lifespan accordingly but I haven't seen a great need yet.

Another inherent limit that the system imposes is on mailboxes themselves. Popular mailboxes such as joe@mailinator.com and bob@mailinator.com get much more email than average. Every inbox is limited to only 10 emails. Thus, popular boxes inherently limit themselves on the amount of email they can occupy in the pool. Use of popular inboxes is discouraged anyway and generally become the creme de la cesspool of spam.

Two more
memory conserving issues is that no incoming email can be over 100k and all attachments are immediately discarded. That latter feature was in years ago but obviously really ruins this whole new wave of image spam (if you see a few seemingly "empty" emails in some popular boxes, they might have been image spam that got their images thrown away).

Spam and Survival

I'd like to emphasize here that Mailinator's mission is NOT to filter spam. If you want penis enlargement or sheep-of-the-month club emails, that's pretty much what Mailinator is good for. We are clear in the FAQ. Mailinator provides pretty good anonymity - but we do NOT guarantee it. We also do NOT guarantee ANY privacy. Its really easier that way for us. Still, it does a pretty damn good job even so. We might log you (used to and it might get turned on again someday, never know) and we DO respond to subpoenas (that whole "jail" thing is a strong motivator).

So, in essence I have no real interest in filtering out spam. I do however, have a great deal of interest in keeping Mailinator alive. And spammers have this nasty habit of sending Mailinator so much crap that this can be an issue. So - Mailinator has a simple rule. If you do anything (spammer or not) that starts affecting the system - your emails will be refused and you may be locked out.

In the new system I created a data structure I call an AgingHashmap. It is, as it indicates a hashmap (String->int) that has elements that "age".

The first type of spammer I encountered was one machine blasting me with thousands of emails. So, now, every time an email arrives, its senders IP is put into an AgingHashmap with a counter of 1. If that IP does not send us anymore email for (let's say) a minute, then that entry automatically leaves the AgingHashmap. But, let's say that IP address sends us another email 2 seconds later. We then find the first entry in the AgingHashmap and increase that counter to 2. If we see another email from that IP, it goes to 3 and so on. Eventually, when that counter reaches some threshold we ban all emails from that IP for some amount of time.

We can put this in words as so (values are examples):
If any IP address sends us 20 emails in 2 minutes, we will ban all email from that IP address for 5 minutes. Or more precisely, we will ban all email from that IP until it stops trying to send to us for at least 5 minutes.

This is really what the AgingHashmap is good for. We can setup some parameters and detect frequency of some input, then cause a ban on that input. If some IP address sends us email every second for 100 days straight, we'll ban (or throw away) every last email after the first 20.

Here's a graph of an average 24 hours of banned IP address emails. Notice at 10am and 11am some joker (i.e., some single IP address) sent us over 19000 emails per hour.



I do have some code that has Java talk back to unix's iptables system to do very hard blocking of IP addresses but its not on right now. Partially because there's no need (yet) and partially because I like to see the stats.

The funny part of this is the error Mailinator gives. Remember the "User Unknown"? Once an IP address is banned and then it tries to open a new connection it will send the SMTP greeting of "HELO". Mailinator will then reply "User Unknown" and close the connection. Of course, it didn't even get the username yet.

Zombies

The next problem came from zombie
networks. Now we were getting spam from thousands of different IPs all sending the same message. We could no longer key in on IP address. As a layer of defense past IP we created an AgingHashmap based on subjects. That is, if we get (again, example numbers) something like 20 emails with the same subject within 2 minutes, all emails with that subject are then banned for 1 hour.

Here's a similar graph. Keep in mind these emails got past the IP filter - so basically they are "same subject" emails from many disparate sources.



You could argue we should ban them forever, but then we'd have to keep track of them and the Mailinator system is inherently transient. Forgetting is core to what it does. This blocking is more expensive than IPs as comparing subjects can be costly. And of course, we have to have enough of a conversation with the sending server to actually get the subject.

Pottymouth!

Finally, we ran into some issues on emails that just weren't cool. As I said, I'm far more interested in keeping Mailinator alive than blocking out your favorite porn newsletter. But, some unhappy people used Mailinator for some really not happy purposes. Simply put, as a last layer, subjects are searched for words that indicate hate or crimes or just downright nastiness.

Boing

Another major influx that happened early on was a plethora of bounce messages. Now thats sort of odd isn't it? I mean Mailinator doesn't send email. In fact, it CAN'T send email so how could it get bounce messages? Well, some spammy type folks thought it'd be neat to send out spam from their servers using forged Mailinator addresses as a return address. Thus when those emails bounced, the bounce came here.

What's worse, is I still get email from people who think Mailinator sent them
spam. Its very frustrating to defend myself against people who are ignorant of how email works ready to crucify me for sending them spam (especially ironic is that I run a free, anti-spam website). As I've said in my FAQ - please feel free to add mailinator.com to the tippy tippy tippy top of your spam blacklists. If you EVER get an email from mailinator.com, its a forged spam.

The good news is that bounces are very easy to detect, and are really the first line of our defense. Bouncing SMTP servers aren't particularly evil, they're just doing their job so when I say "user unknown" they believe me and go away.

On an abstract level, here is what happens to an email as it enters the system.



(and to be fair, there might just be another layer or two thats not on that diagram!)

Anti-Spam revolt

There are 2 more, somewhat conflicting features of the Mailinator server that should be noted. For one, its a clear fact that when we're busy, we're busy. An easy DoS against us would be to open a socket to our server and leave it open. This is an inherent vulnerability in any server (maybe especially multithreaded servers). So, as a basic idea Mailinator closes all connections if they are silent for more than a second or two. Actually, the amount of time is variable (read below). Clearly, we are DoS'able by sending us many many connections, but this blocks at least one trivial way of bringing us down.

Secondly, although we demand servers talking to us are very speedy. We reserve the right to be very NOT speedy. Here's the logic. When Mailinator is not terribly busy, we still demand responses quickly, but we give responses slowly. In fact, the less busy we are, the slower we give responses. It is possible that sending an email into the Mailinator SMTP server could take a very long time (like 10 or 20 or 30 seconds) even for a very small amount of data.

Why? Well.. think about it. Let's say you're spamming. You want to send out a zillion emails as fast as possible. You want every receiving SMTP server to get your email, deliver it to the poor sod who wants (or doesn't want) weener enlargement and then close the connection so you can go on to the next. If you encounter some darn SMTP server that takes 20 seconds to receive your email, the speed at which you can send out your emails diminishes. You might just even think about avoiding such SMTP servers.

It might be a pipe dream to think this is slowing down any spammers, but this does tend to keep my quieter times lasting longer. And it doesn't really hurt me - or my users. And if we eventually get terribly busy, those delays are scaled down to make sure we don't lose any emails.

Sites will ban it

Every time I read some comment about Mailinator, someone always points out something like "Yeah, well sites will start banning any email addressed to Mailinator and then it will be worthless". Guys. Its been 3 years. A handful of sites have indeed blocked Mailinator, but my user base and the number of read emails has only gone up. Clearly, people are finding Mailinator more useful than ever.

I have added at times additional domains (like sogetthis.com and fakeinformation.com) that point to mailinator. Often if a site bans mailinator.com proper, you can use one of those to same effect.

Overall

Many copycat sites have appeared over the years which is pretty reasonable. This idea itself is obvious. The only real hurdle was that it seemed impossible to do given the amount of useless email you'd get. But the copycats had the advantage of seeing that Mailinator actually does work, so they knew what to shoot for. Only a few post their daily email numbers but I've yet to see any that come close to mailinator's incoming email (not that this is necessarily a good thing). I also see that many are using an architecture similar to Mailinator's original which is just fine so long as they either don't get any massive increases in email or are happy to keep buying bigger hardware.

Overall, Mailinator has been a great experience. It was a terribly fun exercise in optimization, security, and generally making things work. Thousands of people use it everyday and its amazing how many people know about it when it comes up in conversation. I've thought many times about how to make a business around it, and there is always an angle, but I've just been to busy with other things.

My hope is that its useful for you and that you tell your friends.


Digg!






Note: Eternal thanks to Jack Lawrence of Syracuse, NY who, in a drunken stupor gave me the core idea (story here), Nicci Gabriel of www.sideofsauce.com for the seriously cool web design, and to Brian Pipa of www.candyaddict.com who, as a big fan of Mailinator, added the very cool "Spam Map" and the RSS feeds.

Monday, December 04, 2006

Review: 2 Books on Anti-Religion

I finished Dawkin's The God Delusion and Harris' Letter to a Christian Nation in the past few weeks.

Its really interesting to finally see a contingient of humanity stand up against religion. That's interesting because its basically fighting for nothing. These atheists (although Harris is quick to point out that "atheist" is a silly word, read below) aren't trying to convert you per se, they're trying to tell you that your religion is pretty silly.

Both agree that we are ALL atheists when it comes to some religions. For example, most of us are atheists to Zeus nowadays (sorry dude). In fact, everyone is probably an athiest to hundreds of different religions. Just pure atheists go one religion further and actually believe in none. They go on with the idea that the word 'athiest' is rather silly in that we don't often categorized un-members of something (although I can think of a counter-example in the word "unemployed").

I absolutely loved Dawkin's earlier books The Blind Watchmaker and The Selfish Gene. The biggest fault of the God Delusion for me was that Dawkins tends to get angry at times. I understand that in his videos of talking to indignant religious people (youtube on "root of all evil"), but there doesnt seem to be much purpose for it in a book. It was an unwelcome distraction (reminded me of Aronson's The Social Animal where he kept being overly politically correct with facts like "men often have more muscle mass than women - Not to say women can't be strong!!! its just that..")

Letter to a Christian Nation is all of 80 pages and can be read in one bus ride (QED). Its clean and concise and worth it given the modest investment.

Both books bring up plenty of Bible contradictions (they both focused on only the Bible it seems) and meta-views of the universe (e.g. if one believes the universe is so complex as to have only been possibly created by a creator, then how was the yet-even-far-more-complex creator created?).

I don't expect either of these book to change many minds given that I doubt many people who are religious will read them. Of decent value is Dawkins discussion on the misconceptions of stem cell research and evolution. He says that many religious people tell him that the universe (or the eye, or the cardio-vascular system, or any other sufficiently complex system) could not have developed by accident. To which he points out as the biggest misunderstanding of evolution.

That evolution is never by accident. Its by massively parallel trial and error (Note that genetic algorithms in artificial intellgence work this way, a million computers (or simulated computers) try a million different slightly different attacks at a problem, the best approach is kept and forms the basis for the next million approaches. All losers are unmercifully deleted). That "part" of an eye is truly an advantage over your neighbor that has no eye at all. And that if your other neighbor has even a better part, his vision gives him an advantage over you. In other words, the eye did not evolve overnite. It began as something that could barely detect light and dark, and slowly got better and better as it evolved to provide more advantage to its owner.

Anyway, these books are just some more fun examples of world religions. Dawkins tells one story of describing an aboriginal religion that involved witches that fly in the night and shoot poison darts at bad people (i.e., "sinners"). To which a priest at his table laughed at what nonsense that religion was. To which Dawkins basically replied that the priest's religion didn't make much more sense itself (plenty of tense moments therein).

The overtones are of course that more people die and more wars are fought for religion than anything else. That God sure wants us to kill each other it seems. Both authors would be happier if religion simply didn't exist. Overall, there isn't any terribly unobvious stuff in there, but both are good reads.

Thursday, November 09, 2006

You think YOU get a lot of Spam?

A month or 3 back, Mailinator was averaging about 1 million emails per day. Now its running pretty solidly at 2.5million+ (see the front page sidebar for the realtime number). Spammers are really going wild.

One peak I happen to notice was 171,000 in an hour, that's ~47 emails per second. Not bad for one little AMD 2Ghz server considering it also handles many thousand web and rss requests too. (who says Java is slow?)

I rewrote the whole system from scratch a year ago (SMTP server and all) as using stock linux services (sendmail, apache, mysql, etc) started dying at less than 1million a day. Monitoring tells me there's plenty of room left before I need to think about another server. Keep em' coming.

I know the spam surge has been reported elsewhere, but consider this my own empirical evidence.




Digg!

Tuesday, November 07, 2006

Evaluating Start-up Ideas - (part TWO)

My entry last week on evaluating start-up ideas netted me some nice feedback. Rereading it, I realize there were a small pile of things that also should be on that list. So, welcome to part 2.

1) Let ideas gestate.

Every new idea seems to be the greatest idea I ever had. Usually it goes something like:

Time 0: get new idea
1 minutes later: (some details)
2 minutes later: start thinking about suit I'll wear when I get to ring closing bell when we IPO

After getting way too excited about way too many ideas, I set myself a rule. Think through an idea for 3 days before I tell a soul. My excitement is usually killing me the first day or so, but by the third day its rather died down. Or, if it isn't, I get a better sense of whether I'm really onto something. Simply put, ideas always look better the fresher they are. You're looking for ones that look good even when not fresh.

2) Consider the size of your market.

If you read Guy Kawasaki's blog you'll see him mention quite often how new companies pitch to him about the immense size of their new market. Guy often chides them for their optimism but even he knows you better believe you do have a big market (and maybe even puff the number in conversations) or you can forget it. Numbers in the hundreds of millions or billions are good - you're really only realistically hoping to glean a smidgen of that.

Consumer markets can be drastically immense. If you're one guy in your basement with an idea that will capture you a 40billion dollar market, that's dandy. Hopefully you have no misconceptions that you'll just own the market yourself (where there's money, there's competitors, and they'll steal plenty, if not most of it before you can penetrate it all) but, the bigger the market, the easier it is to find customers.

Niche markets like hardware stores, developers, and/or mailmen don't suck - so long as you can effectively penetrate most of it - and fast. Simple rule, bigger is better. If big enough (and your idea does actually penetrate it), you'll capture the attention of someone big who will want it for themselves. That either means buying you or chasing you - where the actual outcome happens in the execution (not the idea).


3) "Building a business around a new developer tool" is wrong on so many levels.

Some markets are not only niche - they live in the land of the free. And I don't mean the USA - I mean a market where people are used to getting everything for free.

The software developer inside me keeps coming up with ideas for software development tools. Occupational hazard I suppose. I understand the domain space and understand how to solve the problem. However, development tools are, in general, a really bad place to find good business ideas. Consider that things like Netbeans and Eclipse that took thousands of person-hours to create are given away FREE. Therein lies the rub, developers have created their own culture of giving away tools for free. There is certainly nothing wrong with that, unless of course you think you're going to make a business out of it.

Even then there are niches inside dev tools that are worse than others. Two dollars and an idea for the greatest Java software dev tool ever will buy you a cup of coffee. Java developers are especially used to getting tools for free. In contrast, Microsoft has created a nice culture of getting .NET and windows developers used to paying for stuff. In other words, if you "must" create a developer tool that you intend to sell, stick with Microsoft or some other market segment where paying is an accepted idea.


4) Ideas really aren't worth all you think they are.

Yes I know you're smart. Yes, you amazingly, super-duper wonderfully creative. But in reality, very few ideas are truly novel.

What happens is that two independent technology evolutions (say, the internet and mobile phones) eventually progress to a point where it takes a simple idea to act as a bridge and converge the two. And a product is born. The farther ahead you see that convergence, the more "brilliant" your idea is.

But with every passing day, the distance between the two technologies lessens and your brilliant idea becomes ever so slightly more obvious. In other words, if someone else hasn't thought of it, they will soon. And as more people realize it, the odds of it hitting someone who's looking to start a company increase.

Basically, the winner will be the company that gets things to market fast, creates a marketing and sales infrastructure, and actually makes sales. Don't rely solely on an idea to make it all the way. It has to be supported. If it isnt - at a minimum - it will be stolen (at least in a market sense). You can definitely create a world-class company on a good idea and a great infrastructure. Visa-versa is nowhere near as easy.

And, as an aside, once the technology progressions converge by themselves (apart from your idea) - your idea will probably be made irrelevant (hopefully you cashed out long before then).

5) Competition is good.

If you don't have competition, you don't have an idea. Competition tells you and investors that your idea isn't wacky. If you work 3 years on a product with no challengers, maybe they know something about the day you're going to release and try to make sales that you don't.

Don't be afraid to chase an existing idea if you have what you consider a sub-idea that makes a key difference. Both Altavista and Google did search. Altavista was doing it long before Google started and probably laughed when they heard some challenger Google was going to take them on. I hate to say it but the concept of "stealing ideas" in business is very hard to define. Patents tried to enforce this notion but its so broken (although you still need them of course) that it hardly matters.

Every idea you have is already an extension of some existing idea (Every web 2.0 idea "assumes" the internet "exists". Every "mashup" not only assumes, but steals functionality from 2 or more existing web services). How close your new idea is to old ideas defines how many people will say you stole it. And there will *always* be people saying that.

As a concrete example, I created Mailinator in July 2003. It was first disposable email service that allowed incoming emails to create an inbox as they arrived. There are now a half-dozen copycat sites (some even stole my FAQ questions and license!). Does this matter? Could I do anything about it if it did? Should I try to squash them?

A better tactic is to simply out-market them and introduce new features. Protecting an idea nearly impossible, working to continually better serve your customers isn't. (And, by the way, the idea for Mailinator wasn't mine. It was Jack's).

--

So there's 5 more. I have a feeling I have another handful brewing, stay tuned.




Digg!

Thursday, November 02, 2006

Have a great startup idea? Hmm. Maybe not.

Over the years I've started a small handful of Internet/software projects/companies. Examples include Mailinator, Preemptive Solutions, Inc., and Classhat. Actually, I've started a large handful but no one knows about most of them because they were (in no particular order): dumb ideas, unsuccessful, too hard for me to complete. Given that I now rate any new idea I get according to a set of rules that helps me filter out good ideas from bad. At least, whatever I consider bad. Keep in mind these rules are for the canonical one or 2-person pre-startup - if you have 8million in VC, there's a lot of other magic you can do.

Here they are:

1) If there is no business model, its a hobby, not an idea. I love compiler optimizations. I wrote a Java optimizer soon after Java came out. I spent months trying to figure out how to turn it into a business. But guess what, people don't pay for optimizers, or compilers, or even runtimes. At least not without a strong sales team telling they need that. By and large most ideas I get are about things that I'd love to work on but have no real business model (ala my Classhat project took several years and is, absolutely, a hobby). There's nothing wrong with hobbies, as long as you know what they are.

2) The best ideas make your customers money. If your idea can say "If a customer uses our product, they will make X% more money" (where X is a positive number, even if quite small) - you have won the game. Importantly - I did not say the customer will save X% more money. I said they'll make it. That's a big difference. Saving money is great, but you are then faced with the mission of convincing your customer that if they spend $100k on your product now, it will pay itself back in 8 months. It's way way way easier to say "Use our product and you make 2% more money (of which we get a cut). Don't use it, and don't." Who wouldn't buy that?

3) The best place to be that I know of is B2B2C. That is, you want to be a business that serves businesses that serve consumers. If you're B2C, then welcome to some important challenges. One is to get people to pay for your stuff, which in this Internet world, we're not all that happy about doing. Secondly, welcome to support hell. Its very hard to provide consumer support (and you see many complaints across the net). It takes a lot of support people and a lot of money to do it right, which is why it rarely is. If your idea plans to charge consumers, I'd definitely think twice unless you can ramp up a support system fast. Thirdly, you'll need a powerful infrastructure (apart from support) just to handle large numbers of small transactions. Its harder to sell 1000 $10 widgets than 10 $1000 ones.

On the other hand, if you're simply B2B, life isn't so bad. Big ticket sales but encompassing the market is harder. Sales cycles are long. Again, you might need a faster ramp-up of a sales and marketing infrastructure (which you are going to need eventually anyway), but you're probably ok if the idea is generally good.

4) If you're going B2C, look for revenue models that don't come right from the consumer. Given the last point you're probably thinking I'm crazy given things like youtube or myspace or facebook or google. I'm not (at least I don't think I am). All those places dodged the problems of providing support and tracking sales by giving away their services for free and making money on the backside (whatever that is). Often that's advertising revenue. Often its partnership deals. Simply put there absolutely is an Eyeball Business Model. If you can get the eyeballs, you can sell them. Just try to do that instead of charging them directly. They'll be ornery about it and demand support.

5) Revisit every bad idea every once in awhile. Why is AJAX hot? Because its enabling things technically that weren't enabled before. Are any of these mashups really novel ideas? Not usually. Old ideas become new again by new technology, faster computers, etc.

When the computer game Doom came out it included very little novel technology. All the 3D math and graphics tech had existed for years. The guys at ID were just the first ones to realize that personal computers (as opposed to graphics workstations) at the time had finally gotten powerful enough to do it all in realtime.

Every time bandwidth gets faster for cheaper, previously bad old ideas become new and shiny (e.g., video over the Internet). Mobile phones seem really ripe. Apps and usability currently sort of sucks. Thats really hampering a ton of uses. With every new advancement however, its going to open up new doors.

6) Do your best to create a system of recurring revenue. Advertising on websites and such is easy. But things like packaged software is hard. Why does Microsoft change the doc format every time it releases a new Microsoft Word. Surely it probably includes new features, but why can't old versions just ignore those and load the file (which is really text right?) anyway? Because they don't want it to. They want you to buy a new version of Word to get all the new features (even if you just write letters to your Aunt Edna and would never use them). They want recurring revenue.

Doing that with packaged software is harder and often somewhat transparently greedy. Heard of "software as a service"? That is of course, really, "software that we can just keep charging for". Bottom line is if tires, light bulbs, or razors never wore out, the economics of those businesses would be radically different. Getting more money from your customers (hopefully while providing value to justify it) is a good way to go.

If have an idea that follows #2 above (making your customer money), do your best to simply take a small cut. Small cuts add up and the customer has no risk in trying your product.

Edit: There is now a part TWO to this article.





Digg!

Wednesday, July 05, 2006

Me on VH1

If you've known me awhile you know I was a prime interviewee on a VH1 special called "My coolest years" in 2004. Well, a rerun is airing next wednesday if you never caught it (at 2am - thus set your TIVO).

http://www.vh1.com/shows/dyn/my_coolest_years/series.jhtml

I look rather, um, different now.

Tuesday, July 04, 2006

Paul-Hiking

I currently live in San Francisco and I think its pretty clear to me now that I live in the wrong part of the county. I resisted that idea for awhile - overall, I think my views are relatively liberal although I definitely don't subscribe to any particular political party. I just believe what I believe and I brought these beliefs with me when I moved - this place didn't change me, at least as far as that goes.

The problem here is not that I don't want to belong. I do. I mean at least to some reasonable level. The problem is really that, San Francisco won't have me. People I know here all seem to do pretty much the same set of hobbies and I'm not really into any of them. I guess I never noticed what sort of hobbies people did when I lived elsewhere - but here they seem dangerously non-unique.

Let's see - there's bike-riding, yoga (by the way, you can't just "do yoga", apparently you need to go to a class - either you're an instructor or a student, but you have to be in a class. Also mind you, I don't do/study/teach yoga but I should really start. Seems like an incredibly great way to meet women), rock-climbing, hitting clubs (no surprise), owning pets (which in this city seems like a terrible idea), snowboarding, skiing, and of course - hiking.

All this sounds a bit reasonable - at least mostly. But as the prime example, that last one sorta gets me. Growing up in Ohio, hiking meant simply "walking in the woods" which is all well and fine but by no means exciting. When you're 12 looking for something to do and you are attempting to articulate the idea of heading into the forest with your friend, the last damn thing you'd say is "let's go hiking". Forget that noise. You needed purpose. You needed - adventure.

You'd get far better response with equivalent-meaning yet far more eloquent statements such as "lets go find snakes and blow them up with firecrackers" or "lets go climb trees above a trail, wait for someone to walk by and whip rocks at them". Now we're talking.

Now many people around here go on and on about hiking. And never once do they mention snakes, firecrackers, or whipping rocks. They really just talk about, um, "walking". And maybe shoes (or boots I guess) and GPSes (GPSi?), and like water bottles. What the hell? Who the hell TALKS about water bottles for real? Now mind you, these same people keep inviting me. They keep saying "come hiking! We'll get like sunburnt and drink WATER when we're thirsty out of this neato bottle".

At first thought I can't imagine a reason I'd want to buy overpriced boots and walk around in the woods trying to get bit by ticks. But I think I figured them out. Obviously, no one in their right mind could get truly excited about just "walking around in the woods". There had to be a catch. A gimmick as it were.

And I'm pretty sure I have found the deeper meaning. Given the amount of fervor, enthusiasm, and sheer excitement in their voices when they talk about it - "hiking" is really some sort of subversive code word for "mid-trail ORGIES". Yep. I didn't believe it at first neither.

And apparently, the first rule of Hiking is that you DON'T talk about Hiking. At least not "real" Hiking. You use code like "Boy, this GPS really knows where we're going" (which means like "Orgy ahead") and "Man, this water is sure cold" (which is slang for like "Orgy ahead"). Yeah. right. uhuh.

The problem with it all is that you're not allowed into the real Hiking club just willy-nilly. I have no idea what it takes to get in, but I know for a fact I haven't yet met the requirements. Everytime I go, no one talks about it. No one looks excited. Instead of Hiking (as in the wild orgy type Hiking), the whole group knows we're just going what I call "Paul-Hiking".

Paul-Hiking is like Hiking minus the orgies. The formula is simple - Paul goes Hiking - you walk in the woods as tick food. Paul doesn't go Hiking - wild, crazed orgy parties at trail-marker #7. Its a damn conspiracy.

I'm thinking that the Hiking society only likes to allow the perseverant and or silly people into their orgies. Basically, they make you walk in the woods in the hot sun getting bit by ticks and NOT being allowed to blow-up snakes over and over and over again until you cross some magic threshold and they finally they say "Ok. You're cool. We were just kidding about all that silly walking in the woods stuff. Here, have an orgy".

Funny enough I've been able to apply this "Paul-" prefix to lots of stuff. In fact, I've become wise to it - anytime people get REALLY excited over something that makes ZERO sense to me - yep - its a "Paul-" thing. In effect, if I don't show up - its going to be FREAKIN cool. If I go, its the normal silly version that no one in their right mind could get excited about except to use as a facade for when I'm there.

Examples:

Yoga:
- why would you do this again?
Paul-yoga: Masochistic pain-infliction via bending
Regular-yoga: Some sorta orgies fer sure.

Sushi:
- um.. yeah. Good idea - lets start a restaurant, skimp on oven costs and call it some neato Japanese name! Riiiiight.
Paul-Sushi: Raw Fish
Regular Sushi: um.. I'm not sure, but it sure as hell isn't raw fish. (and probably does have orgies somewhere in there).

Owning a cat:
- Yes. Lets get a skiddish creature with knives on the end of its limbs and let it loose amongst fine furniture
Paul-Owning-a-Cat: A box of shit in your house
Regular Owning-a-Cat: No idea. Maybe they talk or like can make "real sushi" for you or fetch the paper or something. Not exactly sure except that non-Paul-owning cats crap in toilets just like humans and non-Paul-owning hamsters do.

I'm on to them though. On our hikes I keep looking for trail turns we "don't" take or signs etched in the sand like "Orgy. Thataway ->". So far, they've thwarted me good - but I am persistent. Soon enough, I'll be a full fledged member of the secret Hiking (and yoga) club and I won't have to put up with Paul-Hiking anymore. Now where'd I put that "GEE-PEE-ESS" *wink*.

Wednesday, February 08, 2006

India Jack

My friend Jack and his GF Jo are on a 6month trip to India and has been writing me emails that I find downright funny, sad, and at time enlightening. But mostly funny.

Installment one:

------------------------------------

Dude!

I'm in, like, India and shit.

If New York is 'The City that never Sleeps', then New Delhi is 'The City that never Shuts The Hell Up'.

Been here a while, all is well. No vomitting or anal squirting at all, though some brat on our bus to Udaipur did both in the aisle. Jo had a bout of Not Feel Well, but it passed.

Internet is hard to find and shitty when you do.

Finished our tour of Rajasthan. Rode a camel for 3 days. Yikes. Slept in the desert. Sweet. 4 days in Jodhpur, yes that's where the pants come from.

In Bombay now. Just had a cup of Monsoon Malabar coffee. They make it just down the coast. It was a swanky coffee shop, so a fatty cup cost rupees35 (about 80 cents!). Down in Kerala it'll be more like rupees 10 or 12. It's only about 85 degrees here, so everyone is wearing sweaters. Our cab driver said it was cold and so I told him that where I live it is now about -5 C. and there's 40 inches of snow so far this year. He looked at me like I might bury an axe in his head and then take his daughters.

Heading to Gokarn tonight. Christmas on the beach. We may even get a hotel room


Cheers,

Jack

Tuesday, February 07, 2006

Whoever....

Whoever came up with the name "Attention Deficit Hyperactivity Disorder" was just downright mean. That'd be like if there was a disease for people that mumbled and they named it "Moomummenners disease".

Sunday, February 05, 2006

Bah.




This always happens - I get something cool and people start copying me. Especially Brad Pitt - he pisses me off. (First that whole "dating Angelina Jolie" thing - now this).

Brad, I see you got a black one .. um.. yeah, don't worry thats pretty cool too (sorta).

Thursday, February 02, 2006

Introducing .... Yapdog

Well.. not having a car has started to be a minor annoyance. I toyed with idea of getting a junker - but I'm sort of done with junkers. So then I decided to look at cars I like (i.e. M3, etc) but honestly it would be a complete waste. I need a car at most once or twice a week, it'd be a shame to let a nice car rot in the garage most of the time.




Thus - I decided to get a bike. Chances are you've never heard of the brand - it's a MV Agusta Brutale 750. Simply put - very italian and rather ornary.

I've been riding since I was 8 and this will be my 4th street bike, although its my first non-enduro. Nicci and I were heavily in the habit of naming our vehicles (VW = spike, last bike = skiddy, SLK = slik/silky, saab = zap, mini we never got = dink).



With that in mind, hence I christen "Yapdog". Its been pointed out to me that this is a wimpy name for a mean looking bike. Actually, that's the point. Simply put, tough guys can wear pink because after all, they're tough guys.

New Blog Format

Hail Loyal Readers !!! (both of you!)

I've decided to turn this into more of a "regular blog" - as in tell you what I had for breakfast, what I named my goldfish, and other innane boring type things.

What of these lengthy, ranty and sometimes irreverant articles? I'm moving them to my alter ego's blog at iggychaos.blogspot.com.

So - check here if you want to read about the new socks I bought - but check the iggychaos for the articles you've come to know.

Saturday, January 07, 2006

Damn you Lizard Brain!

Lately I've been having a nasty fight with my lizard brain. Come to think of it, I've been fighting with that idiot as long as I can remember. I hate that guy. This might sound funny to you if you're not familiar with the concept, but I promise, you too know your lizard brain all too well. I can prove it too, and I bet you hate yours like i hate mine.

Plenty of modern research suggests that our brain is made up of layers that perform many different functions (surely, I'm being simplistic on a complex topic, see Jeff Hawkin's "On Intelligence" for a far more informative discussion on specifics). In short, our brains share many characteristics as brains of lesser creatures. Different sections of our brains have different functions and some of our more primal instincts are controlled but what some call the "Lizard Brain" (implying we share this brain structure with lizards). Now mind you, the way I use the term, I am considering it to have a definition only I have provided. Again, if you're really interested in a better treatise, they are out there. I'm far more concerned with how this primal part of my brain makes me act rather than its exact anatomy.

Ok, so there's the oversimplified technical overview which probably doesn't clear much up. Allow me to give you a in-your-face introduction to what I'm talking about.

At one point in my adult life I went on a stunningly successful diet. At least it was stunningly successful in the "crash" sense - I lost more weight than ever before but it didn't last long. That's probably typical but my current point. I am 5'11" and after age 20 I have weighed between 156 and 206 pounds. The result of this diet was the 156. What was the secret to this super diet? I'll tell you (move your ear closer to the screen).. here it is... (don't tell no one, it's a secret) - it's simply: "don't eat".

Ah, you say, you've heard that before. But when I say "don't eat", I'm not kidding. And everyone knows that *saying* "don't eat" is about 1000 times easier than *doing* "don't eat". Let's be real, its pretty damn easy to make diet resolutions walking out of a restaurant. It's a heck of a lot harder to do it. In fact, that's the real secret - the way I made it happen.

It was really just a wild extension of a simple rule - if you have a bundt cake sitting on your kitchen table, eventually you're going to eat it. Thusly, I made sure that I had no bundt.

I learned this very important rule without knowing I learned it. Once lizard brain is hungry and sees a bundt cake, he is relentless - and he is really, really good at being relentless.

Lizard: ...zzzzZZZZZ...
You: "Huh, I still have that bundt cake Aunt Jane forced on me"
Lizard: "*cough* wha?"
You: "Um. nothing."
<*noises of Lizard brain processing the idea of bundt cake*>
Lizard: "Eat Bundt"
You: "No"
Lizard: "Eat Bundt"
Lizard: "Eat Bundt"
You: "No, I'm strong"
Lizard: "Eat Bundt"
Lizard: "Eat Bundt"
You: "Hmm, whats on tv?"
Lizard: "Eat Bundt"
Lizard: "Eat Bundt"
Lizard: "Scratch itch"
<*scratching*>
Lizard: "Eat Bundt"
Lizard: "Eat Bundt"
You: "Ooh. Reality TV about losing weight"
Lizard: "Eat Bundt"
Lizard: "Eat Bundt"
<*channel change*>
Lizard: "Eat Bundt"
Lizard: "Eat Bundt"
You: "Woah. Gwenneth Paltrow bikini special!"
Lizard: "Eat Gwenneth Paltrow"
Lizard: "Eat Gwenneth Paltrow"
You: "as if"
Lizard: "ok, Eat Bundt"
Lizard: "Eat Bundt"
Lizard: "Eat Bundt"
You: "Hmm. Maybe just a little bite"
Lizard: "Eat More Bundt"
Lizard: "Eat More Bundt"
You: "Damn, might as well have a whole piece now"
Lizard: "Eat Bundt"

Lizard is as dumb as they come, but he is persistent beyond all imagination.

Just like in the movies he'll be that little devil on your shoulder telling you the wrong thing to do. You'd damn well think that if you had a teeny-tiny angel on one shoulder and a stinky little devil on the other, you'd have NO problem figuring out who to listen to. Yet as my admission of 206lbs sits above, I was a bit fuzzy on the concept.

Truth is - its not a little devil. Its lizard brain (bastage).

Your perfectly rational mind can take days or weeks of consideration about how you'll plan your diet so long as the lizard stays asleep. Keep in mind, he is a benevolent master. Lizard brain is directly tied into your pleasure centers and he's damn well not afraid to use them. You give him strawberry cheesecake, he gives you seratonin (i.e., brain candy). Its a fair deal.

What I learned is that much like a real lizard, my lizard brain had absolutely no foreplanning. No concept of the future at all. He was ALL about the here and now. If my tummy was full he disappeared. Hell, maybe he went off to some lizard strip club smoking lizard cigars or something. Heck if I know. And heck if I care.

But - I used this time to scheme against him. In fact, he had so little perception that LATER he might want to lather my ass in bundt cake and Gwenny Paltrow bikini pictures that I could make it well-known inside my mind that I was plotting his demise - and in a blank stare of overconfidence, he just sat there in the back of my mind indifferently flipping through some lizard magaizine.

Aha! I have you my scaly friend!

I removed from my house all interesting food. Actually - more precisely, I simply stopped stocking my house with interesting food. Lizard helped me clear the house over a few days.

Then, I stopped carrying my ATM card (lizzie is dumb - but even he quickly realized that money can equal food). Eventually going to lunch at work daily, I ran out of money. Now I could have always driven home, got my ATM card, grabbed some cash, drove to McWendyKings's and got a Tripple-cow burger with all the fixins, but that was WAY too much work. Despite how bad lizard wanted that burger, he simply couldn't navigate motivating me through all those steps. At one point he had me sifting the seat cushions of my car (I swear, it was him - not me) for quarters and dimes to add up to 75 cents to buy a Rally burger. Let me tell you, Lizard is a powerful god. Its pretty scary to consider your personal resolve as you weave through a drive-thru to order a 2-ounce lipmeat-burger and pay in dimes.

But eventually, the dimes ran out. Now remember, we don't call him lizard brain for nothing. Lizards run through life following a pretty simple program:

1) If See bug, Eat bug - goto 1
2) move a random distance or wait a bit
3) goto 1

Thats it. Once there are no more bugs (i.e., or in our context we'd say dimes, bundt cakes or rally burgers) he does not lament the point. He simply waits for the next opportunity and really pulls the pressure off you. No need to have you scurrying about wasting energy trying to catch flies when there are none.

The simple fact is that you have the ability to plan ahead. Lizard does not. In fact, things like artificial sweeteners or contraception are commercial ventures at lizard brain deception. Both of these things are in some respects made for lizzie. They make him think he is getting what he wants without really giving it to him. Plain and simple, he's an idiot. He's happy as a pig in mud thinking that sucralose is sugar. You can even tell him to his face its not sugar - but lets be real, he's a friggin lizard - he doesn't know better.

I'm not saying this battle is won. I'm not even saying this battle can be won. Everyone has weak moments (probably even lizard). When you're weak and he is strong, you're probably gonna bundt.

In addition, I must admit - his existence is probably part of why life is fun. I love a good challenge and lizard is always there to provide me with one. Damn you Lizard brain! I shall defy thee! Well, right after I eat these cheetoes I will.