Obviously everything I say is personal experience and opinion. I think the layoffs were long overdue and should've happened sooner. There was a general understanding between my coworkers and I that Uber definitely over hired in 2016. Thanks to that a lot of engineers ran out of things to do which led to political infighting over roadmap, ridiculous redundant resourcing - every single team had mobile/web, severe shortage on infra teams since head count was already taken by main Uber teams. We even started building and maintaining our own chat app. I disagree that all the cool eng project that we put out and share here is a waste of time or resources. Those project was initiated by real needs on Uber and only the best make it out into the rest of the world. Engineers that shipped those projects are still here and we would've done it regardless of whether the 435 people that were hired in the first spot. I fully realize that it's an insensitive thing to say but I think it's important that it's expressed somewhere.
I think the lesson here is to grow responsibily, and it's real people's lives that are being affected. Don't hire people to alleviate your current engineer's burnout and stress. Learn to plan better and to prioritize aggressively instead of just hiring and getting everything done.
God I love the Not Invented Here disease. Somebody, somewhere, thought "hey, slack is way to expensive, we don't want to get 'locked in', and 'mumble mumble privacy cloud evil stallman' so lets write our own chat application.... how hard can it be?"
Boom, now the company is straddled by a shitty, homebrew chat application that is maintained by nobody because the original author left for greener pastures. Nobody dares replace it though because that would be sacrilege. That chat app is part of the company, after all!
Arg.... I hate, hate, hate when engineers with a bad understanding of business and too much time on their hands lock their organisations into homebrew crap that has nothing to do with the value delivered by the business.
Like I completely agree with your overall point about reckless overengineering, but you also just listed three true and really good reasons to avoid integrating Slack.
No reason to use Slack when such options exist, but also no reason to write your own chat app, either.
This is valid not just for Mattermost but many other apps that support sending e-mails. And that's why it's important that instead of giving up and outsourcing it, we all try to set it up and have first-hand experience in challenges we have to face in the process. Because if everybody just uses Mailgun & SES, in a decade or two it won't be be possible to set up your e-mail server anymore (of course you will be able to, but your e-mails won't reach most customers).
Moreover, email is hard and does require a dedicated team, but the order of magnitude is wrong here. 5 people is plenty.
Adding to other large company problems, these people will need management layers, the team will need a charter for growth. Who wants to be the maintainer of the Mattermost servers in a large company? Add all of this up and then the slack deal starts to look reasonable.
Edit: Just to add some real numbers, slack costs $12.50 per user per month  - that is ~$750k per year for 5000 users = 5 engineers. (More like 3 really in the Bay area)
If it takes you that many engineers you are either using the wrong software or using it incorrectly.
Compare with a small development team for a significant percentage of their time to build a halfway decent chat tool.
Miles of difference.
Given that Slack can be seen as IRC with some pretty decoration glued up to an Elasticsearch instance, and that Uber already has a highly trained tech org in place, if it wants to make a Slack "clone" for less than, say, $25million (cost of Slack to Uber for 5 years), it could actually be a pretty sensible decision- especially when you take into account the tax and stock-price benefits of capital expenditure.
Maybe they realised it was a superior solution to building their own app.
Uber's internal chat tool is built on top of Mattermost.
Anyway, as another commenter pointed out, on an enterprise scale I'm sure you can get a good deal. I can also imagine however that at that scale you'll want multiple Slack servers / instances - which would add cost given how slack charges per person, per server.
So there's a few alternatives then; self-hosted (some parties will offer enterprise customers the option to host an instance of their application on the customers' internal hardware), or rebuild it yourself. The latter one however takes a bit of math - how much is it going to cost to build, maintain and run it? It's easily underestimated.
And yet, it's also easy for people working in an engineering culture with nearly limitless engineering capacity to just build it themselves - motivations being "how hard can it be", and "I need something more challenging than tweaking this button for my employer to earn 0.1% more money on this particular feature". I've seen it happen far too often.
> Slack began as an internal tool for Stewart Butterfield's company Tiny Speck during the development of Glitch, a defunct online game. "Slack" is an acronym for "Searchable Log of All Conversation and Knowledge.
His first game company also didn't do well, so they pivoted to release Flickr.
Look at their competitors in Asia (all-in-one services like WeChat, GoJek etc. where search, chat, ride hailing, banking etc. are all built in one app) and it would make sense for them to have an engineering team building a chat app.
Even the public kubernetes workspace has almost 75k people in a single channel.
What what does scale mean anyway? Just employees using it or employees + contractors?
It's easy to armchair quarterback on this stuff, but I find it hilarious how tech companies that employee supposedly smart people justify writing this kind of stuff. I'm sure some junior engineer did a cost/benefit that completely neglected the total cost of ownership for writing a chat app and focused on the sticker shock of having to pay tens of thousands a month for a chat app.
I've worked in plenty of companies with engineers who think they are legal, privacy and security experts. If we made business decisions based on these folks, we'd never do anything. Hell, every damn text change we makes invokes a few who worry about the legal ramifications ("should we file a JIRA ticket with legal?").
Sadly without somebody stepping in, people actually listen to them instead of the actual experts employed by the company... thus you wind up wasting tons of engineering bandwidth building a feature / platform / product that has no business existing all (or worse, not building some feature / platform / product) because nobody bothered to check with the legal / privacy / security team that some concern by a engineer was actually a genuine concern.
And even then, in any good organization legal / privacy / security teams exist only to point out all the risks in any business decision... sometimes it is worth the risk despite what legal / privacy / security say. What appetite the company has for the risk depends on a lot of things that are well outside of legal/privacy/security.
the general counsel and thus the entire legal org required for any company approved comms system a way to automatically by default destroy chat logs after 7 days (and to turn that off for folks who were on legal holds). slack in 2016 did not have that capability.
Booo on legal.... like I said earlier, it's easy to armchair quarterback. Sometimes I wonder how the fuck humanity even manages to make progress with so much waste and boneheaded decision making.
Running your own DC is the same as building your own chat app. It is almost always more expensive and almost all the people who claim they can do it cheaper inhouse aren't factoring in all the costs--both costs that are easy to measure and those that are almost impossible to measure
Examples of hard-to-quantify costs for running your own infrastructure include:
- What is the cost to the company of being three versions behind on mongo because IT doesn't have the bandwidth to upgrade? How many work-arounds do teams have to do because they couldn't use the latest features of mongo?
- What is the cost to the company when developers cannot quickly spin up new environments like they could using a service like heroku?
- How much innovation and new business (read $$$$) is not happening because the in-house IT simply doesn't have the toolset so a dev can quickly riff on an idea?
- How many engineers have good ideas and don't bother building them because spinning up an environment is too much trouble?
I'd argue by not using AWS you are holding your product teams back and stifle innovation. Unless of course you want to waste company money to build out your own half-assed provisioning tools that are lousy knock-offs of AWS's tools. But then, you might as well just use AWS...
What I was trying to point out is someone engineered a system that costs 8M/mo to run yet only handles 1M rides per day.
That doesn't really sound that bad, considering that rides can easily cost dozens of dollars. Just about any other non-tech business would kill for such low overhead.
Other non-tech businesses, hell EVERY business that I've ever worked in that used AWS did better per transaction - healthcare, digital advertising, virtual office tooling, gaming. If you can't do better than a credit card processing fee per paid customer interaction across the company on AWS, you might as well be selling retail goods...and depending on age, your company might be soon.
Companies like Lyft are doing a lot more than updating a few fields for each ride. They would be processing massive amounts of location data and they are building out models to eventually use for self driving cars.
It is sizable, not ridiculous. And optimizing your cloud cost is definitely your own responsibility and deserves every bit of attention.
Which is even more scary. Why isn't uber laying all these people off and paying cash money for a real chat application? What competitive advantage does uber have maintaining a chat application?
You can take your hands off completely, even go and do something else, read a book, write an email, book some flights, and the chat will drive itself ... somewhere.
I am talking about Samsung and Apple here..
Except Snapchat, I still can't work out what that's about.
The equal and opposite problem is managers who don't understand the sunk cost fallacy and refuse to throw out home grown solutions. Sometimes there might have been a good idea to homebrew a thing years ago because nothing good existed. Now several better solutions exist but the company refuses to adopt them because doing so would mean "throwing away" years of work, and the managers who signed off on that work are afraid they'll look bad.
I have the impression that Slack has gotten worse. We're again having problems of notifications not showing up.
Yes, really, for business. Look past the theming.
Discord has a much more solid tech base (and engineering team) behind it than Slack, IMHO. A better API, too; and—unlike Slack—real support for third-party clients!
I built my own niche team-collaboration app back in 2012, and I tried out a lot of things, but ended up building my whole own stack. That was back then. If I was doing it again today? I'd just make it an alternative Discord frontend.
(There are also offerings specifically designed for the "build your own custom frontend for our chat backend" use-case, like https://pusher.com/chatkit, but IMHO Discord even has these beat.)
You don't have to make those links (and you can turn off, per server, the ability to create those links.) You can invite specific users. The user already has to have a Discord account, though.
And that's the thing; Discord has a different accounts model. Which is part of what I was referring to when I said they had a more solid tech base: when you're signed into a dozen Slack workspaces, the Slack client has to keep a dozen websockets open, because the open connection is for a given Slack account's view of the world, and Slack accounts are a sub-resource of Slack teams. This isn't a scalable model, and when the Slack app is using 1.2GB of memory and two full CPU cores, this is much of the reason why. When you're signed into a Discord account, on the other hand, you just have one connection to the server—and one state-sync session that includes all your open channels in all those servers—because your profile on a given Discord server is a sub-resource of your global Discord account.
It's the difference between how an email client connects to N accounts through IMAP, and how the Gmail mobile app is connected to N GSuite accounts.
And, as such, it wouldn't really make sense to have Discord SSO at the account level, because a Discord account is associated with the individual, not with a particular employee record of a particular Enterprise. It's not like an email address; it's more like a Keybase identity or a Facebook profile. (You could certainly have particular profiles under the Discord account that did SSO to a particular server, but this wouldn't be traditional SSO-as-we-know-it; it'd be more of a "connector" flow, like when an app wants to use your Github profile.)
Github is a good comparison, actually. Most corporations just use Github, not Github EE. You don't Enterprise-SSO to Github.com. Github.com is an SSO provider, in fact, that other sites (e.g. Asana) can take auth from!
As for github.com, our official IT recommendation that people don’t use their personal accounts, but create one just for work. This way there is no worries about personal data there.
(This is at a subsidiary of a large international company. I can believe small startups use a simpler methods)
I'm quite fine with that, frankly. Highlighting correctly requires actually parsing the file. I use vim and the colors get wonky way too many times, because it tries to "keep it fast".
Bridging in Slack, Gitter, IRC and other platforms makes working with external teams a breeze, everything is in one place.
Since when did wanting to be a software developer mean there’s “one true way” to do software dev?
Why are developers supposed to be budget babysitters for a huge company with tons of people watching budgets?
I don’t owe my employer being an expert in all contexts to the benefit of their business. I am not their servant. I write code to serve contexts they want served.
Perhaps the memo didn't make it around to everyone.
Well that's the incentive for that kind of behaviour, pad your resume and get another job.
> hey, slack is way to expensive, we don't want to get 'locked in', and 'mumble mumble privacy cloud evil stallman'
gets followed up immediately (almost inevitably, in my experience) with
> so lets write our own chat application.... how hard can it be?"
...rather than, say, "hey, I googled and there are a few FOSS alternatives for 'group chat with history': Zulip, for one; Mattermost, for another. Let's see if either of those fit our needs. If they don't quite, though, let's also see if either one of them has a solid, customizable codebase to fix up!"
And even then they could have just used Matrix, which ticks all of those boxes.
The real issue here is a lack of vision at the top. The current CEO needs to step down for the health of the company. He has a clear lack of vision for the company, as evidenced by his statement of "do your divisions look like what they should if we had to start over?" Who says that? Something so vague that is basically grasping at straws and shows a complete lack of strategy. Would Jeff Bezos say something like that? I highly doubt it, because he has complete control over his company and a vision that has worked at micro and macro levels.
You can do things that aren't directly related to your core business and these things should generate value both internally and externally. If you can't do this, eventually, your company will stop growing.
This where understanding business matters. AWS makes sense as a business and it benefits mostly from economies of scale which is something unique that a massive company like Amazon can provide the service which few other companies could.
NIH stuff is also rarely externalized outside of the business. Even as open source. Which as the OP pointed out needs proper investment of human capital. Not some side project of a bored engineer or two.
Most innovation happens outside of mega companies by small teams for a good reason. Even the Skunkworks type approach is only useful for certain types of work like Googles X projects. Some B2B SaaS programs roll out of parent organiations but typically they are started by teams who quit the bigger company to solve the problem they had.
The first AWS service was launched over 15 years after Amazon.com started selling books on the Internet. It wasn’t even a separate brand until after Amazon S3, SQS and EC2 were launched.
Close. It was roughly 11 years. I've been using Amazon AWS since early 2007; I believe it launched to the public in the form we know today in early 2006. They started selling books online to the public in mid 1995.
If you go back to the very first AWS services - 2002 - it's closer to being only seven years.
If a different company had dominated the on-demand computing space before Amazon’s peak load issues became too big a problem, they very may as well have gone with that and not made AWS.
Almost nobody needs to write their own chat app. Slack is fine.
Look familiar? https://mattermost.com/
I often wonder how accurate the roadmap prioritization can be. I do agree it’s pretty silly to build a lot of infrastructure that will never effect your clients.
If you have for instance a billing process that needs to be automated, custom infrastructure is needed. Manual approvals of all transactions is a lot even for banks. Operations can always benefit from something that cuts their overhead time down a significant chunk. Operations approval on transactions seems like an operations problem but clients having to wait for approval means now the problem is shifted.
Disclaimer: I work on the M3DB team.
There was some earlier discussion here on why trust between employees and employers is so low now, and this is a great example. Employees being laid off while new positions open up for that exact same role so that they can hire new developers at a lower cost or avoid having to promote other employees.