Usenet is just one example of a much bigger trend of the last twenty years: the Net—standardized protocols with multiple interoperable open-source clients and servers, and services being offered either for money or freely—being replaced with the Web—proprietary services locking in your data, letting you talk only to other people who use that same service, forbidding client software modifications, and being ad-supported.
Instant messaging with multi-protocol clients and some open protocols was replaced by many tens of incompatible services, from Google Talk to Whatsapp. Software telephony (VOIP) and videoconferencing, which had some initial success with free services (Jingle, the SIP standards) was replaced by the likes of Skype. Group chat (IRC) has been mostly displaced by services like Slack.
There are many stories like these, and many more examples I could give for each story. The common theme isn’t that the open, interoperable solution used to rule these markets—they didn’t always. It’s that they used to exist, and now they almost never do.
Explaining why this happened is hard. There are various theories but I don’t know if any of them is generally accepted as the single main cause. Maybe there are a lot of things all pushing in the same direction. Here are a few hypotheses:
Open protocols don’t have a corporate owner, so they don’t have a company investing a lot of money in getting people to use them, so they lose out. For-profits don’t invest in open protocol-based services because people can’t be convinced to pay for any service, so the only business model around is ad-based web clients. And an ad-based service can’t allow an open protocol, because if I write an open source client for it, it won’t show the service provider’s ads. (Usenet service used to cost money, sometimes as part of your ISP package.)
The killer feature of any communications network is who you can talk to. With proprietary networks this means who else made the same choice as you; this naturally leads to winner takes all situations, and the winner is incentivized to remain proprietary so it can’t be challenged. Interoperable solutions can’t compete because the proprietary providers will be able to talk to interoperable users and their own proprietary users, but not vice versa.
In the 80s and early 90s, when the first versions of crucial protocols like email were created, the Net was small and populated by smart technical people who cared about each others’ welfare and designed good protocols for everyone’s benefit—and were capable of identifying and choosing good programs to use. Today, the Web has three to four orders of magnitude more users (and a similar increase in programmers), and they aren’t any more technologically savvy, intelligent or altruistic than the general population. Somewhere along the way, better-marketed solutions started reliably winning out over solutions with superior technology, features and UX. Today, objective product quality and market success may be completely uncorrelated.
There are other possibilities, too, which I don’t have the time to note right now. This is late in the night for me, so I apologize if this comment is a bit incoherent.
the Net—standardized protocols with multiple interoperable open-source clients and servers, and services being offered either for money or freely—being replaced with the Web
The Web, of course, if nothing but a standardized protocol with multiple interoperable open-source clients and servers, and services being offered either for money or freely. I am not sure why would you want a lot of different protocols.
The net’s big thing is that it’s dumb and all the intelligence is at the endpoints (compare to the telephone network). The web keeps that vital feature.
proprietary services locking in your data, letting you talk only to other people who use that same service, forbidding client software modifications, and being ad-supported.
That’s not a feature of the web as opposed to the ’net. That’s business practices and they are indifferent to what your underlying protocol is. For example you mention VOIP and that’s not the “web”.
and now they almost never do.
Never do? Really? I think you’re overreaching in a major way. Nothing happened to the two biggies—HTTP and email. There are incompatible chat networks? So what, big deal...
And an ad-based service can’t allow an open protocol
Sigh. HTTP? An ad-based service would prefer a welded-shut client, but in practice the great majority of ads are displayed in browsers which are perfectly capable of using ad-blockers. Somehow Google survives.
Explaining why this happened is hard.
No, not really. Here: people like money. Also: people are willing to invest money (which can be converted into time and effort) if they think it will make them more money. TANSTAAFL and all that...
The Web, of course, if nothing but a standardized protocol with multiple interoperable open-source clients and servers, and services being offered either for money or freely. I am not sure why would you want a lot of different protocols.
This is like asking why, before HTTP, we needed different protocols for email and IRC and usenet, when we already had standardized TCP underneath. HTTP is an agnostic communication protocol like TCP, not an application protocol like email.
The application-level service exposed by modern websites is very rarely—and never unintentionally or ‘by default’ - a standardized (i.e. documented) protocol. You can’t realistically write a new client for Facebook, and even if you did, it would break every other week as Facebook changed their site.
I use the example of Facebook advisedly. They expose a limited API, which deliberately doesn’t include all the bits they don’t want you to use (like Messenger), and is further restricted by TOS which explicitly forbids clients that would replace a major part of Facebook itself.
The net’s big thing is that it’s dumb and all the intelligence is at the endpoints (compare to the telephone network). The web keeps that vital feature.
That’s true. But another vital feature of the net is that most traffic runs over standardized, open protocols.
Imagine a world where nothing was standardized above the IP layer, or even merely nothing about UDP, TCP and ICMP. No DNS, email, NFS, SSH, LDAP, none of the literally thousands of open protocols that make the Net as we know it work. Just proprietary applications, each of which can only talk to itself. That’s the world of the web applications.
(Not web content, which is a good concept, with hyperlinks and so forth, but dynamic web applications like facebook or gmail.)
That’s not a feature of the web as opposed to the ’net. That’s business practices and they are indifferent to what your underlying protocol is. For example you mention VOIP and that’s not the “web”.
I mentioned VOIP exactly because I was talking about a more general process, of which the Web—or rather modern web apps—is only one example.
The business practice of ad-driven revenue cares about your underlying protocol. It requires restricting the user’s control over their experience—similarly to DRM—because few users would willingly choose to see ads if there was a simple switch in the client software to turn them off. And that’s what would happen with an open protocol with competing open source clients.
Never do? Really? I think you’re overreaching in a major way. Nothing happened to the two biggies—HTTP and email. There are incompatible chat networks? So what, big deal...
Email is pretty much the only survivor (despite inroads by webmail services). That’s why I said “almost” never do. And HTTP isn’t an application protocol. Can you think of any example other than email?
Sigh. HTTP? An ad-based service would prefer a welded-shut client, but in practice the great majority of ads are displayed in browsers which are perfectly capable of using ad-blockers. Somehow Google survives.
Google survives because the great majority of people don’t use ad blockers. Smaller sites don’t always survive and many of them are now installing ad blocker blockers. Many people have been predicting the implosion of a supposed ad revenue bubble for many years now; I don’t have an opinion on the subject, but it clearly hasn’t happened yet.
people like money. Also: people are willing to invest money (which can be converted into time and effort) if they think it will make them more money. TANSTAAFL and all that...
That doesn’t explain the shift over time from business models where users paid for service, to ad-supported revenue. On the other hand, if you can explain that shift, then it predicts that ad-supported services will eschew open protocols.
HTTP is an agnostic communication protocol like TCP, not an application protocol like email.
Huh? HTTP is certainly an application protocol: you have a web client talking to a web server. The application delivers web pages to the client. It is by no means an “agnostic” protocol. You can, of course, use it to deliver binary blobs, but so can email.
The thing is, because the web ate everything, we’re just moving one meta level up. You can argue that HTTP is supplanting TCP/IP and a browser is supplanting OS. We’re building layers upon layers matryoshka-style. But that’s a bigger and a different discussion than talking about interoperability. HTTP is still an open protocol with open-source implementations available at both ends.
It requires restricting the user’s control over their experience
You are very persistently ignoring reality. The great majority of ads are delivered in browsers which are NOT restricting the “user’s control over their experience” and which are freely available as “competing open source clients”.
Can you think of any example other than email?
Sure. FTP for example.
Smaller sites don’t always survive
Why is that a problem? If they can’t survive they shouldn’t.
That doesn’t explain the shift over time from business models where users paid for service
The before-the-web internet did not have a business model where users paid for service. It pretty much had no business model at all.
Huh? HTTP is certainly an application protocol: you have a web client talking to a web server. The application delivers web pages to the client. It is by no mean an “agnostic” protocol. You can, of course, use it to deliver binary blobs, but so can email.
HTTP is used for many things, many of them unrelated to the Web. Due to its popularity, a great many things have been built on top of it.
The point I was making is this: when a server exposes an HTTP API, that API is the protocol, and HTTP is a transport just like TCP underneath it and TLS on top of it. The equivalent of a protocol like SMTP on top of TCP is a documented API on top of HTTP. The use of different terms confused this conversation.
But that’s a bigger and a different discussion than talking about interoperability. HTTP is still an open protocol with open-source implementations available at both ends.
My point is, you can’t interoperate with Facebook or Gmail or Reddit just by implementing HTTP; you need to implement an API or “protocol” to talk to them. And if they don’t have one—either deliberately, or because their HTTP traffic just wasn’t designed for interoperability—then there is no open “protocol”.
The great majority of ads are delivered in browsers which are NOT restricting the “user’s control over their experience” and which are freely available as “competing open source clients”.
The great majority of web ads are actually displayed, keeping revenue flowing. They’re not removed by adblockers. Even if everyone installed adblockers, I believe ads would win the ensuing arms race. It’s much easier to scramble software, mix the ads with the rest of the web site so they can’t be removed without breaking the whole site, than to write a program to unscramble all websites. (In contrast to the problem of unscrambling a specific given website, which is pretty easy—as shown by the history of software copy protection.)
That is only true because the server provides the client software, i.e. the website’s client components. That the browsers are open source is as irrelevant as that the client OS is. The actual application that’s trying to enforce showing the ads is the website that runs in the browser.
FTP for example.
The amount of use FTP sees today is completely negligible compared to its market share twenty years ago. But I agree, file servers (and p2p file sharing) are good examples of an area where most protocols are documented and interoperable. (Although in the case of SMB/CIFS, they were documented only after twenty years of hard struggle.)
Why is that a problem? If they can’t survive they shouldn’t.
I didn’t say it was a problem…
The before-the-web internet did not have a business model where users paid for service. It pretty much had no business model at all.
That’s not true. There were many services being offered for money. Usenet was one. Email was another, before the advent of ad-supported webmail.
Usenet is just one example of a much bigger trend of the last twenty years: the Net—standardized protocols with multiple interoperable open-source clients and servers, and services being offered either for money or freely—being replaced with the Web—proprietary services locking in your data, letting you talk only to other people who use that same service, forbidding client software modifications, and being ad-supported.
Instant messaging with multi-protocol clients and some open protocols was replaced by many tens of incompatible services, from Google Talk to Whatsapp. Software telephony (VOIP) and videoconferencing, which had some initial success with free services (Jingle, the SIP standards) was replaced by the likes of Skype. Group chat (IRC) has been mostly displaced by services like Slack.
There are many stories like these, and many more examples I could give for each story. The common theme isn’t that the open, interoperable solution used to rule these markets—they didn’t always. It’s that they used to exist, and now they almost never do.
Explaining why this happened is hard. There are various theories but I don’t know if any of them is generally accepted as the single main cause. Maybe there are a lot of things all pushing in the same direction. Here are a few hypotheses:
Open protocols don’t have a corporate owner, so they don’t have a company investing a lot of money in getting people to use them, so they lose out. For-profits don’t invest in open protocol-based services because people can’t be convinced to pay for any service, so the only business model around is ad-based web clients. And an ad-based service can’t allow an open protocol, because if I write an open source client for it, it won’t show the service provider’s ads. (Usenet service used to cost money, sometimes as part of your ISP package.)
The killer feature of any communications network is who you can talk to. With proprietary networks this means who else made the same choice as you; this naturally leads to winner takes all situations, and the winner is incentivized to remain proprietary so it can’t be challenged. Interoperable solutions can’t compete because the proprietary providers will be able to talk to interoperable users and their own proprietary users, but not vice versa.
In the 80s and early 90s, when the first versions of crucial protocols like email were created, the Net was small and populated by smart technical people who cared about each others’ welfare and designed good protocols for everyone’s benefit—and were capable of identifying and choosing good programs to use. Today, the Web has three to four orders of magnitude more users (and a similar increase in programmers), and they aren’t any more technologically savvy, intelligent or altruistic than the general population. Somewhere along the way, better-marketed solutions started reliably winning out over solutions with superior technology, features and UX. Today, objective product quality and market success may be completely uncorrelated.
There are other possibilities, too, which I don’t have the time to note right now. This is late in the night for me, so I apologize if this comment is a bit incoherent.
The Web, of course, if nothing but a standardized protocol with multiple interoperable open-source clients and servers, and services being offered either for money or freely. I am not sure why would you want a lot of different protocols.
The net’s big thing is that it’s dumb and all the intelligence is at the endpoints (compare to the telephone network). The web keeps that vital feature.
That’s not a feature of the web as opposed to the ’net. That’s business practices and they are indifferent to what your underlying protocol is. For example you mention VOIP and that’s not the “web”.
Never do? Really? I think you’re overreaching in a major way. Nothing happened to the two biggies—HTTP and email. There are incompatible chat networks? So what, big deal...
Sigh. HTTP? An ad-based service would prefer a welded-shut client, but in practice the great majority of ads are displayed in browsers which are perfectly capable of using ad-blockers. Somehow Google survives.
No, not really. Here: people like money. Also: people are willing to invest money (which can be converted into time and effort) if they think it will make them more money. TANSTAAFL and all that...
This is like asking why, before HTTP, we needed different protocols for email and IRC and usenet, when we already had standardized TCP underneath. HTTP is an agnostic communication protocol like TCP, not an application protocol like email.
The application-level service exposed by modern websites is very rarely—and never unintentionally or ‘by default’ - a standardized (i.e. documented) protocol. You can’t realistically write a new client for Facebook, and even if you did, it would break every other week as Facebook changed their site.
I use the example of Facebook advisedly. They expose a limited API, which deliberately doesn’t include all the bits they don’t want you to use (like Messenger), and is further restricted by TOS which explicitly forbids clients that would replace a major part of Facebook itself.
That’s true. But another vital feature of the net is that most traffic runs over standardized, open protocols.
Imagine a world where nothing was standardized above the IP layer, or even merely nothing about UDP, TCP and ICMP. No DNS, email, NFS, SSH, LDAP, none of the literally thousands of open protocols that make the Net as we know it work. Just proprietary applications, each of which can only talk to itself. That’s the world of the web applications.
(Not web content, which is a good concept, with hyperlinks and so forth, but dynamic web applications like facebook or gmail.)
I mentioned VOIP exactly because I was talking about a more general process, of which the Web—or rather modern web apps—is only one example.
The business practice of ad-driven revenue cares about your underlying protocol. It requires restricting the user’s control over their experience—similarly to DRM—because few users would willingly choose to see ads if there was a simple switch in the client software to turn them off. And that’s what would happen with an open protocol with competing open source clients.
Email is pretty much the only survivor (despite inroads by webmail services). That’s why I said “almost” never do. And HTTP isn’t an application protocol. Can you think of any example other than email?
Google survives because the great majority of people don’t use ad blockers. Smaller sites don’t always survive and many of them are now installing ad blocker blockers. Many people have been predicting the implosion of a supposed ad revenue bubble for many years now; I don’t have an opinion on the subject, but it clearly hasn’t happened yet.
That doesn’t explain the shift over time from business models where users paid for service, to ad-supported revenue. On the other hand, if you can explain that shift, then it predicts that ad-supported services will eschew open protocols.
Huh? HTTP is certainly an application protocol: you have a web client talking to a web server. The application delivers web pages to the client. It is by no means an “agnostic” protocol. You can, of course, use it to deliver binary blobs, but so can email.
The thing is, because the web ate everything, we’re just moving one meta level up. You can argue that HTTP is supplanting TCP/IP and a browser is supplanting OS. We’re building layers upon layers matryoshka-style. But that’s a bigger and a different discussion than talking about interoperability. HTTP is still an open protocol with open-source implementations available at both ends.
You are very persistently ignoring reality. The great majority of ads are delivered in browsers which are NOT restricting the “user’s control over their experience” and which are freely available as “competing open source clients”.
Sure. FTP for example.
Why is that a problem? If they can’t survive they shouldn’t.
The before-the-web internet did not have a business model where users paid for service. It pretty much had no business model at all.
HTTP is used for many things, many of them unrelated to the Web. Due to its popularity, a great many things have been built on top of it.
The point I was making is this: when a server exposes an HTTP API, that API is the protocol, and HTTP is a transport just like TCP underneath it and TLS on top of it. The equivalent of a protocol like SMTP on top of TCP is a documented API on top of HTTP. The use of different terms confused this conversation.
My point is, you can’t interoperate with Facebook or Gmail or Reddit just by implementing HTTP; you need to implement an API or “protocol” to talk to them. And if they don’t have one—either deliberately, or because their HTTP traffic just wasn’t designed for interoperability—then there is no open “protocol”.
The great majority of web ads are actually displayed, keeping revenue flowing. They’re not removed by adblockers. Even if everyone installed adblockers, I believe ads would win the ensuing arms race. It’s much easier to scramble software, mix the ads with the rest of the web site so they can’t be removed without breaking the whole site, than to write a program to unscramble all websites. (In contrast to the problem of unscrambling a specific given website, which is pretty easy—as shown by the history of software copy protection.)
That is only true because the server provides the client software, i.e. the website’s client components. That the browsers are open source is as irrelevant as that the client OS is. The actual application that’s trying to enforce showing the ads is the website that runs in the browser.
The amount of use FTP sees today is completely negligible compared to its market share twenty years ago. But I agree, file servers (and p2p file sharing) are good examples of an area where most protocols are documented and interoperable. (Although in the case of SMB/CIFS, they were documented only after twenty years of hard struggle.)
I didn’t say it was a problem…
That’s not true. There were many services being offered for money. Usenet was one. Email was another, before the advent of ad-supported webmail.