The lost art of telnet


A long time ago, on an operating system far,
far away...




It was a period of protocol war.
Raw text connections sent
over the internet had
gained a strong following.

During this time, greater
needs and complexity arose
eventually leading to the
wide adoption of the hyper
text transport protocol

Knowledge of how the web worked
on these lower levels soon
vanished. It is up to you to
learn the old ways and restore
freedom to the internet...



There's something we once did back in the early days of the wild wild web, back before HTTP was the only game in town. It was called telneting, now known only in passing to some programmers, and a hand full of old school hackers that might still use it.

We lost something when we stopped using telnet, we stopped understanding how protocols are higher abstractions of raw text, how fragile things are, how simple they might be.

What is telnet?

Okay, so I'll admit telnet is actually the wrong technical term for it, but it's how a lot of early hackers learnt about protocols and did neat things. For all intents and purposes we're talking about sending raw text over TCP/IP. To be totally honest, this is a bit about internet protocols.

Fun and games

One of the coolest things I remember back even during the turn of the millennium was something called Multi-User-*, where the * was a place holder for "created Kingdoms" or "Dungeons". They were called MUDs, MOOs, MUSHes and MUCKs. They're a dying breed, born in the mid 1970s, but by now with these fancy web scripts and css tricks most people have entirely lost their interest in them.

There were no graphics or special data encodings, just a dumb box of text that you got to play with (for hours and hours). It may be hard to see why people would bother with these things. Eventually a handful of the servers did branch out and add images, but generally they weren't a feature.

I really got into them at an early age because one particular type of server had several programming languages exposed to the player. You could log in, and start writing code that manipulated the game world, and other users could walk into your part of the world and start to play with them. Anything you wanted you could program while playing the game with others, and export that code to another server somewhere. I actually ran one of the most popular sites on something called MPI back when I was about 12 years old, but that's long defunct now. Others played the game for the adventure of other user-created world and plots they wrote (that's right, we had user-created content as the focus of the internet back in the 1970s before I was even born, web2.0 and MMORPGs: go eat your heart out).


Well, wash my arm! Stren Withel will rue the day he eyed me over 
and started flipping tables at me!

It was a great environment for me to learn in, because it was a chat room (like IRC), it was a game world that I could explore and interact with.

The above is a snapshot of the Discworld MUD, where the theme is based on the famous Discworld series by Terry Pratchett! It's been on-line for almost 20 years now, and there are roughly 50-150 players logged in at any time.

The MUCK I was a big part of was actually the NMC project (a fork of a popular MUCK server). The only NMC server around right now as far as I know is Redwall MUCK, which you might enjoy as it has a web-based interface that lets you connect from a browser. One of the really cool things with that MUCK is that Brian Jacques, the series author, gave his permission for it to operate before his unfortunate passing earlier last year.

We don't really have this sort of thing on the Web as we know it. I mean, yeah sure, we have crummy phpBB installs where some people role play or do fantasy football. If you want a cheap laugh you can hop over to Kongregate and play Don't shit your pants, but you know what? It's not quite the same thing.


Sweet, I got a crown after unlocking 
all achievements.


There was something more engaging about using my imagination with 10-150 people, all of us sharing and creating. Somewhere along the line people started enjoying Skinner box rewards from killing sheep in MMORPGs for ten to fifteen bucks each month is more than a game that's text-based, and you have to create your own world and plot. Anyone who's experienced an addiction to NetHack knows just how fun these games can be.

Knowledge

One of the really cool things that existed was the gopher network. Before Wikipedia, we had a number of servers with networks serving to normal gopher clients. We still have a few today.

It sounds a bit silly, but I wish the idea of what gopher was ended up winning -- if it had, we wouldn't be in this "over-design-everything" mess we're in now. You can check out the gopher manifesto over here, which honestly reads more like a post mortem than anything. To summarize both of those, if we had gopher we would have websites that deliver pure content without any of the design chrome, or spam.

Pure media (sounds, videos, graphics, text, whatever) without needing to force a heavy markup layout on us to send the content. It's what the static web needs to be: just the knowledge, and not the sparkling prettiness of bad web design design and spam-filled "news" sites that floods the net today (one paragraph per page with ten ads and irrelevant noise). It's tragic the protocol died so long ago. Had it evolved with the same effort that HTTP had, we may have seen a very different net today, a cleaner one with less spam and marketing adds flying around. Perhaps one of the coolest things: it would have made this whole "mobile site" thing a total non-issue; we're only delivering the content, not the flood of menus, pretty backgrounds, background music, and flashing animated advertisements.

There's still some gopher servers out there, but the unfortunate part is that the technology has remained in near-stasis since 1991, so to get an appreciation for it you need to think you're back on a dial-up modem and JavaScript won't be invented for another four years.

Using lynx to browse a gopher of everyone' favourte web comic.

There are a few knowledge and communication based protocols floating around, like IRC, which has survived very well, and as most technically inclined people have likely used it before, so I won't be going into detail there.

Email is another great one. What could be simpler than SMTP? It has an interactive help system, and you even greet the service with "HELO" and lets you write emails on-the-spot, not a single data burst.

For fun, for profit.

Knowing all of this is interesting, sure, but what do you really get out of it?

Well, for starters, you gain an appreciation for how the internet is not the web, and especially not the web browser... it's so much more, so much richer, and has so much greater potential than the web! If you understand telnet you understand the Internet isn't a magic box that renders HTML strings into pretty pictures, that's only what happens after the internet takes place.

You understand the internet, you understand that every time you open a web address you send raw text that looks like "GET /Resource" at it's very core, there is no magic going on, it's not a closed black box that is beyond your understanding. It's so dead-simple you could make something just like it.

WireShark is a great tool for inspecting network traffic.
This is some of the raw text that was sent when I loaded Reddit.

So why did we suddenly stop writing cool and inventive protocols and stuff everything into an HTTP service? There's a few good reasons. All of these mixed protocols lead to total madness on the web, we didn't really have a common ground to do cool things on, you needed lots of different software to do lots of different protocols.

Of course, the web browser was originally meant to be the common user-interface for many protocols, it ended up being the HTTP/FTP browser. HTTP didn't end up being the best tool for everything but merely a tool for everything. Today we we have the SOAP monster and RESTful API services, along with some strange form of APIs that aren't either of these things but still run if the sun and the moon line up the right way. HTTP is nice in that it's brought a lot of normalization to the internet, but it's also halted innovation from doing something that isn't an HTTP/HTML driven web. Something that you may find entirely amazing is that over twenty years ago some smart people at XeroxPARC were talking about social media and personalized user interfaces: two things that are only recently present on the web. They were talking about doing this on the protocol level with something called WAIS, and if you have an hour you should watch this lecture from 1991 (right when the web started) and ask yourself why has it taken so long to finally get some of things?

We don't always need to write code for the HTTP/HTML internet to do some fun hacking projects, but it never really happens. My guess is that most simply don't realize they can make their own protocols, or cannot see a reason to to replace HTTP to try and accomplish something else in a different way, the same way FTP, Gopher, SMTP, or even IRC do. After all, all of those protocols can be emulated in functionality through HTTP with a RESTful or SOAP API.

Not enough people understand them

Not because they're complicated. As you've seen here, some are dead-simple. They aren't understood because they aren't thought about, because they're out of sight: it's hidden away, tucked inside the internals of your browser, email client, or any other software. It just works. HTTP/HTML was reaching a tipping-point back in 2005 where we had a hard time writing interactive apps like desktop software on the web, but the advantages of the web were promising. If AJAX and JavaScript hadn't come in and matured the user experience of the web as we knew it, we may have very well ended up with a new protocol already that would have done animations and rich interfaces as part of the protocol and encoding.

What am I suggesting? Learn the protocols. I'm willing to write about any specific protocol in more detail if there's a high demand for it. You can write a POP3 email client in less than 250 lines of code.

What would I really like to see? I wish for more hacker-inspired innovation in the protocol arena, where people just invent things only to push boundaries and see how something else might work. I want this for a few reasons. Politicians decided hyper-linking to pirate content is illegal, some decide blocking DNS makes sense, others say encryption is illegal... those of us who understand the internet truthfully understand how asinine these policies are. The other reason I wish to see more playing around is simply so we don't keep working ourselves into one corner of technology. The web is slowly becoming one protocol which was never designed with the entire web in mind. We're creating a feature creep simply because we push so hard to get everything into a browser that only supports HTTP and HTML.

What else might we have if we didn't have the HTML/HTTP web?
  • We already have the ability to build faster websites, if we could only have a protocol that sent pre-computed DOM trees, rather than HTML documents.
  • We could send multiple documents at once-- rather than sending a document that describes more documents to request to make back to the server that tells us what requests we need to make.
  • We could make a protocol specifically for discussion sites, so places like Reddit might exist in Telnet, unwrapped from the chrome of the web, and can work as a messaging service.
  • We could make everything a JSON object, because that's the hip thing right now.
  • We could send content in TeX format.
  • What else would you do?
We don't have new protocols because of the inertia HTTP has built up, but that doesn't mean you, me, and all the enthusiastic hackers around here can't take our internet back from the people who don't understand it.






What's in a name database?

All too often I will see an application somewhere cause users to scream bloody murder because the author didn't know about some of the material I'm about to cover here. These things took a lot of time and reading to fully realize not only because they are not intuitive but also culture dependant. Hopefully you can follow my findings and save yourself the pain of bad design, and the time spent reading about cultural differences.

Many websites ask us for our name, under the assumption that they need this sort of information.Of course they do, how else would they personalize emails to us as if they were our dear friends that care so deeply about us.


Does anyone actually feel special when getting emails from some random forum they posted a question on 5 years ago? I'm glad they reminded me they're #1 on my birthday. Well wash my arm folks, I'm touched!

Nobody on the planet can justify requiring a name for sending these lame emails.

So, what's your name?

Surprisingly, this is actually the wrong question to be asking someone. Let me quickly explain to you what a name really is. In the German language (which is one of the languages that helped form English), you don't normally ask someone "What is your name?" but more literally you will ask them "What are you called?". A name is simply some noises and letters we use as a reference to someone else. Historically, in many cultures it wasn't unheard of to go by different names all at the same time, or at later dates. You may have heard of William Shakespear being referred to as The Bard, or Williams Shatner who goes by Bill with closer friends. Such nick-names are still very common today, perhaps sometimes goofy.

It's not unheard of for people to change their names either, obviously so with married persons in modern western cultures, but even historically thousands of years ago, entire names would change to signify a major life event. Names are not always inherited either, such as the tradition of ancient Hebrew cultures where there was no "last name" or "family name," you were simply David, son of Jesse, or in the Sikh tradition where men will take the last name Singh upon baptism.

The question I'd like to go back to is, why are you asking for this person's name?

What is a name?

A name is a reference to a person relative to another group of people at a specific time. My friends call me BDG and write it on my birthday card, my boss calls me Brian, the government calls me Mr. Graham. If I change my name, my friends will still probably call me BDG, my boss might call me both names, and the government will bill me for the name-change processing fees.

So, again, a name is flexible, it changes, and it can be a number of things, to anyone, at any time. So before you ask someone for their name, you need to tell yourself first why you need their name. That's right, scope out requirements rather than mindlessly collect data.

If you've been paying attention to registration forms you'll see them come in all types. Facebook's changed their signup form and run their big data crunching over names hundreds of times. Twitter's needs are actually very straight forward-- you get a "Full Name", twitter account, email, and password.


Speaking strictly as an end-user for a moment here. I don't know you or your service/company on any personal level, so I don't really trust you. I don't want to tell you anything about me because you're just going to spam me on my birthday with some email. Why you would ever need my name is frankly beyond me. You'd better have a great reason that's valuable to me as an end-user, and not you as an advertising or data-collecting agency.

On sites that feel my "Government of Canada name" is important without providing me a clear reason why simply get a fake name such as "Bruce Oxford". It's going to happen to you when you're only asking for my name so your emails can say "Dear Bruce" in the subject line, rather than a valid reason that I care about. Email greetings generated by a robot are stupid, my name on a parcel is not.

You don't need my name, stop asking for it. Provide me whatever service it is you do and leave me the hell alone after that. Nobody on this planet is naïve enough to think that some website mass email script included me because they want to be my friend, and making money had nothing to do with it.

I actually do need your name.

I saw an email sent out to a travel agency once from their customer. The agency is collecting data used to book an individual flights and hotels. While it might seem obvious to us that he may need to give them his name and address to do things like work on processing entry visa requirements to China, he didn't understand and simply replied with a very short email asking "Are you totally high?! I'm not giving that out!!". Yes, he was an extreme case, but there's something to be learnt here. People care about their information, so explain to them why you need it, even if it's obvious to you. Let them know you're not selling it to some spam farm in Russia for twenty Kopeks.

Ask for their name after you've ensured there's a value to them in having it (and doing mental gymnastics doesn't count, it's a strong sign you need to immediately stop the mistake you're about to throw onto your customers).

Ask them for the name you need. You may need the government name that's the most recognized, or simply a pseudonym like "Incognito". Let them pick something if you don't actually need it for any regulatory purposes, or even better, don't use names at all. My email from something I pay money for didn't include my name and I couldn't be more thrilled about it.



Okay, so we can now ask them for their real name?

You have a real reason now that actually provides value to your users? Of course you can! So make your database tables with your three fields like you were about to.
  • First name
  • Middle name
  • Last name
And now let's have the last German Minister of Defence come and put his name in, so here's his name and start typing it in: Karl Theodor Maria Nikolaus Johann Jacob Philipp Franz Joseph Sylvester Freiherr von und zu Guttenberg. 

Yes, our friend Karl is the exceptional case being of nobility, so please ensure your database cannot accept any members of the bourgeois and stick with the simple traditional proper modern anglo-saxon naming contention of having a first name, middle name, and a last name. Everyone who isn't in this naming scheme is a person who's business you most certainly don't want.

Over in Hungary names are different from how you might expect them to be. If your name is Bob Smith Magyar, in Hungary you would be known as Magyar Bob Smith. Their culture expects the family name to come before the given names. They have similar conventions in many Asian cultures, for instance, my friend Dai Wai (who goes commonly by Dave, and doesn't use Chinese writing over here to write his name, but back home and on his passport he does). In some Malay, Korean and Arab cultures it may be the custom to refer to a person as the parent of their children (rather than the Hebrew tradition "David, son of Jesse", one could expect "David's father").

I'm not the only person saying this stuff either. The W3C has a lengthy document describing how names differ around the world, what the implications are, and how to design for this concept.

Now, if for some reason you actually agree that a naming convention is a silly reason to not just take someone's money they're willing to give you, keep reading.

We can't do three names, I get it. What do I do?

We have two options here, add in at-least fifty more fields to the database to deal with this (by the way, "von und zu Gutenberg" is german for "from and to Gutenberg", so spaces doesn't mean it's another name), or we go back to that question I keep telling you to answer: why do you need this name?

Most of you don't need any names from an end-user. You can work off some random user-handle or something and let us keep what shred of privacy we have left. Just let me change it later if I need to signify a change in my life.

So let's accept there is no such thing in the international world as a "first" name. Hungarian first names are the family name, English first names are the given name. There's commonly a Given Name (that is, the name that was given to you, or you gave yourself), and the Family Name (or more correctly, SurName)

So, ask for, only when needed, a place for the following:
  • Given Names (all of the names you were given)
  • SurName (the name of a family you belong to)
You can't make them required. People living in Turkey, Indonesia, Afghanistan, Tibet,  and south India commonly have no surname at all. Here in Canada, we have a senator named "Nancy Ruth", where Ruth is actually not her surname. Nancy formally renounced her surname of "Jackman", leaving only her given names. This isn't entirely unheard of either, many people all over the globe will renounce their surname on paper.

That's it. Just ask for those two names, and have flags go off it it doesn't meet your guidelines to have someone review it. Let them enter in all the accents, spaces, and other marks they need to.

Don't be dumb

I want to warn you here, because you might not see it elsewhere. I see a lot of people bang out some code for what they think is a suitable program for taking name input, only to have their database crash when Jack O'Brian comes and puts his name in. That's because you don't understand SQL injection, so go figure that out (start searching the web, it's a big topic). Don't pass names through a regex, don't force capitalization or lower case, "McDonald" is a valid name, so is "DeLuca". Don't play around with people's names once they give them to you.

Don't use the same name when testing your database either. Ensure you have fake test data. The last thing you want is a real person's data becoming abused (especially your own) at some point. There's a service that doesn't specialize in edge-cases that may push the limits of what a name is, but an excellent tool for generating large sets of fake persons. It is called the Fake Name Generator. I use it.

One last note about Honorifics.

There's a huge mistake I see all the time, where programmers don't know what Honorifics are. You may have never heard of the word, but you do know some of them.

If you see the name, "Mr. Oxford", it's that "Mr." that is the honorific. If you read through this post and thought names were complicated, you have no idea how bad these get. My advice, much like names,. is avoid dealing with them.

Sometimes, you might need to because of a regulation. My advice in this case is stick with the normal honorific system of your country of operation. If you're in Japan, you shouldn't worry about Mr. Oxford, or if this is Oxford-sama or Oxford-chan or any of the millions of combinations over all of the cultures out there.

The Honorific is truely worse than a name. It will change depending on age, social status, martial status, and perhaps other things. Keep it to a simple list as per the minimum requirement of whatever regulatory need you're working with. If you don't need them, don't do it.

I've seen forms list this as a "salutation" input for some reason, and no, it's an honorific. I understand honorific is a long word, so simply put "title" which is close enough that English readers understand the context.

Tl;dr
  • Don't ever ask me for a name
  • Don't ever ask me for a title
  • Seriously, don't ask.
  • What type of name do you need from me?
  • Okay, explain why you need it and only if you really do need it
  • Support any text as "Given Names" and "Surname". Ideally you support all proper names.
  • Support as few honorific titles are you need to. Ideally you support 0.
  • The scheme for names changes quite frequently through history, and culture.

Labels: , ,







Full list of archived posts