ҹɫֱ²¥

Transcript

Introduction
 
Hi, everyone. Welcome back to another Matt Talks. This week, I wanted to talk about the scalability of our platform and security, which is one of those really complex topics that I can't talk well about, but Ryan can. And that's why I said, Ryan, you come and talk to hoteliers in simple language about the really complex things that you do in our business. But, Ryan, who are you and like, what's your title, and how did you end up at ҹɫֱ²¥?
 
I'll do my best. Yeah. Thank you for having me. So, yeah, my name is Ryan Tomlinson. I'm the VP of Platform Enablement.
 
And there are quite a few responsibilities that I'm looking after. One is our infrastructure, so our cloud infrastructure.
 
Everything that essentially underpins our products keeps them up, stable, reliable for our customers, but also security. So making sure that we have a very high security posture both internally and for our customers. And then everything around data and what we call developer experience, which you can get into later if useful. But we have over a hundred people in platform enablement today.
 
And when you were a young boy, was this the aspiration always to do this special role that you do?
 
I was always torn. So, actually, what I wanted to do was be like a PE teacher. I'm quite into sport and I think running, as you know. So I always, like, younger days, wanted to be a PE teacher. It wasn't till later that I kind of got into computing and, yeah, things like that.
 
So I was always interested, what's the first thing you built yourself on a computer?
 
Oh, that's a very good question.
 
I used to build websites for people. So, like, very early, very young. I can't even remember the age. Maybe teens. But I remember, like, building websites for people just because I found it fun. Like, the instant gratification of being able to build something yourself and yeah. Kind of see it out there, which is kindlike that a little bit now, but.
 
Do you think about AI now where you don't even need to have coding skills to even build a website? And how that is going to transform the space that you're operating in?
 
That's the thing that I think excites me the most. Like, what you can do today with tools like Lovable, and even Cursor and things like that. Like, you can build stuff without knowing how to code. It's gonna be interesting to see how it changes the industry. There's, of course, a lot of rhetoric around, will we need software engineers anymore? Yeah. Which I think we will.
 
I think the role will just change slightly, just to make you become more powerful, almost.
 
Exactly. Exactly. Yeah. So I think, like, ultimately, I see it that everyone will be able to create products, like, even yourself, Matt. Like, if you wanna have an idea, you're no longer trying to articulate that to somebody else what's in your mind. You can build it, and then a software engineer might do the last mile and get it into production. But it's exciting.

Chapter
How to ensure 24/7 availability
 
So what does your team do to make sure that ҹɫֱ²¥ is available, like,24/7, around the clock, around the world, all the time? Like, how do they go about that job?
 
There are quite a few areas. There are quite a few things that we do. Like, we are constantly iterating and trying to improve our infrastructure.
 
Firstly, everything's in Microsoft Azure. So Microsoft's a cloud, and we're in multiple, what they call, availability zones, but you should just kind of think of those as, like, data centers.
 
Right? So at once, we're in multiple data centers, which means if one goes down, which they can, we just switch straight across automatically to the other one. So we are constantly thinking defensively about these scenarios that could occur and how we would respond. We even do drills.
 
I don't know if you know that, Matt, but, like, we actually do incident drills where we say, right, if this happens, how would we respond? The team will kind of rally, and we essentially kind of practice our response.
 
Nice. Do you use like, I've heard of companies that use external hackers to try and constantly hack the infrastructure, but you pay them. Like, these are friendly hackers instead of the unfriendly ones?
 
Yes. We do a couple of things. So we do internal, kind of, resilience, kind of checking our systems. But, yes, we have a bug bounty program, which is kind of what you're describing, where we pay hackers to find breaches in the system potentially or vulnerabilities.
 
Sorry. And we'll pay them depending on the severity. So we actually use a a third party called HackerOne, and they're like an intermediary. And if a hacker finds something, depending on that severity, we pay them a a different a different amount.

Chapter
Understanding planned downtime
 
Nice. Yeah. So, often, I see these emails come in from other platforms, and we're cloud native, but they're also coming from cloud native platforms, where they say, we have a planned downtime on Sunday between five and nine. And I'm like, what? They're gonna just take their whole platform offline? And I just don't understand why this happens, but maybe you can give some light as to why this actually happens for even for cloud systems still.
 
I mean, the reality is there should be no excuse for that these days. Like, there are perhaps some edge cases, but if you build a system that's resilient and built for migrations, like, whatever it might be, you might be moving on to a new database or a new technology.
 
A lot of these companies have got them into a situation where there's so much complexity where they can't do it seamlessly.
 
But you have to build defensively. You have to build your architecture and your infrastructure thinking, in the future, we may wait, want to change this technology or this database.
 
How do we build this in a way that we can do it without impacting customers? So with everything we do, we never want to have any downtime.
 
We actually deploy, I don't know if you know this as well, Matt. We deploy twice a day at the minute, sometimes more. So we're constantly shipping features and new changes to customers and fixes, and nobody notices it. We actually, late last year, changed the whole region.
 
So we moved our hosting from the Netherlands to I remember this.
 
Yeah.
 
Zero downtime. So we literally, no customers were affected at all, and we changed entire countries where our primary hosting was.
 
Like, in the early days of ҹɫֱ²¥, we would have deployment day once per week or once per two weeks. So we'd take, right, on a Tuesday because we could never deploy on a Friday because then, you know, we'd have to bug fix over the weekend. So we'd have, like, a Thursday or Tuesday release day, and we pack up all of these features until that day, and then we would release them. And then we'd spend the next day trying to fix all the boxes that we just released into the environment. We've matured so much since then.
 
So but, like, do we release multiple things at the same time or as and when features come in, we just deploy them? Or, like, what's the process to make sure that we don't deploy badly built features?
 
It's ultimately, it's all about reducing risk and confidence. So we write automated tests. Every feature should have a bunch of tests to check that it actually functions correctly before it actually goes out to a customer. So we have different environments that we test in before it goes to, goes to production.
 
We try and release as much as possible, and this is what's kind of almost counterintuitive, where people think, oh, you shouldn't be doing this many changes.
 
Like, we aim to do fifty plus changes to production because the more you push smaller changes, the less of an impact it's gonna have because you can narrow it down to say, right, just this one little thing went out. If there's an issue, we know it's this one little thing. If you have and we have three hundred engineers, I think, building product today, all working on the same code. So if you batch all of that up, then they have the potential to have a kind of bigger impact or more catastrophic impact if something goes wrong. So our goal is to release even more and faster.
 
I love it.

Chapter
How cloud allows you to scale better
 
So, I've come out of a space in hotels where we used to have a server in the backroom, and we'd have an IT team that would just take care of this server. And the system was always pretty fast. We're now in the cloud, so there's a change in, obviously, having IT teams on-site because you don't necessarily need them for at least not for our product.
 
But it also means that we have to design products in a different way in the cloud. Like, how do you and a lot of hoteliers are still like, I like knowing where my server is. I like that I can lock that door. How do you think about the security and stability of platforms by having a server in your back office?
 
Again, it's about the risk of that. Like, if something goes wrong with that server, which it will. Right? Like, that's computers. They're complex things. They'll go down.
 
There's no redundancy. Right? There's no redundancy outside of that. If you have a power cut or something kind of goes down, so you're really increasing the risk of your business by being located on premise.
 
I remember back in this is, like, fifteen years ago. I worked for a tech company. I used to have to drive down to a data center with a USB stick, and I would plug it in to actually deploy the code and make the changes. Or if we couldn't scale, I'd have to buy a new server, literally drive down for three hours, plug it in, and that's how we scaled. Any issues, again, you had to kinda fix it. The cloud really is just geographic compute. You have, like, servers across the world that within seconds you can pick from.
 
We even have what's called auto scaling. So what you can say is, if we see this much demand, just add more servers. You don't even have to touch it. It just adds more servers automatically.
 
You don't get any of that if you're on premise. Like, you're really increasing the risk to your business and there's really no reason for it these days.
 
Because we today have, what, twelve thousand customers live on the platforms across different servers. We work with Amazon. We work with Microsoft. Does this scale infinitely? Or do you, at some point, have to rebuild everything from the ground up?
 
No. I think so.
 
You build to this to the predicted demand. Right? You don't kind of over-optimize. So whether we double or triple that, it's all possible.
 
Right? Like, we can infinitely scale, but there are things that we have to do to ensure that that is possible. And there are multiple techniques for scaling. You know, there's, like, things called, like, sharding and things like that, but there's multiple ways that you can actually scale.
 
But we work kind of proactively to predict the demand. So if we're gonna add x thousand new properties, we know about it well ahead of time and make sure that there's the headroom in the space that we can actually scale to meet that that demand. But it's a constant focus, right, of the teams, a constant investment. But, essentially, we can infinitely scale the system.
 
Nice. And when something goes wrong, like, and things will go wrong, like, it just happens. At some point, something is gonna go wrong with one of the partners that we work with. I'm assuming you have all kinds of reports. Like, I'm imagining you at your house with a giant screen with all kinds of reports, but I don't know if that's what your house looks like. But how do you go about figuring out what's gone wrong, and how do you solve for that?
 
Well, essentially, every team has that. So you're imagining, right, you know, like the big screen with all the graphs and the dashboards on it. Every team at ҹɫֱ²¥ has that because they need to understand: are they scaling appropriately? Is the performance okay? So we are constantly, it's called observability.
 
So anything that moves in our system, anybody clicks a button, anybody checks a customer in or out, or does a report, we log absolutely everything. That all goes into what we call observability, and we can create dashboards of it. But we're talking literally terabytes a month, which is like huge and huge amounts of data, because when any of it moves, we want to log it and get the insight of it.
 
Because how do we know that you're doing a good job and that your teams are doing a good job? Because, obviously, if the system is up and running, it's like that, then we're like, great. Everything is fine. But, like, it's like, usually, we know that you're getting, doing a good job when something goes wrong. But I actually wanna know in advance whether the platform team is doing great work. So, how do we know whether your team is doing amazing work?
 
We track, like, metrics. We have, like, a scorecard of the metrics that we have. So in terms of performance, we have a set of, like, service level objectives that we expect. So, for example, 99.99 percent is our uptime, service level agreement.
 
But, and I wanna pause that before you continue this. 99.99 percent uptime, most hotels have a night audit, which means your system goes down for, you know, half an hour every single day. Like like,99.99 uptime means there is no night audit. It never goes down. Like, you can always check customers in. It is, like, mind-boggling that there are still hotels around the world that have a thing called the night audit, where you take the whole system offline to figure out how to move to the next day. Like, so I just wanted to pause there before you jump back in there.
 
Yeah. And even though that's our SLA, like, we're, of course, striving for a hundred percent. Anything below that is like a disappointment to us. So we're constantly measuring the performance of the site, you know, how much headroom we've got, as I said before, how's the uptime. So we're constantly measuring and scrutinizing these metrics literally on a daily basis.

Chapter
Enterprise readiness of ҹɫֱ²¥
 
So would you say, so ҹɫֱ²¥, we've come up through small independent hotels and then slowly moved up to larger hotels. Would you say ҹɫֱ²¥ is enterprise-ready, and what does that mean to you?
 
I think, like, enterprise-ready is a funny one, this one, because it's like, I expect I have a high bar, and we all do at ҹɫֱ²¥. So that we have the same expectations for all of our customers, regardless of whether you're a kind of small independent or you're a large, enterprise chain. So, really, for me, it's about stability, scalability, compliance, but that bar should apply to absolutely, everybody.
 
But, of course, we do. Right? We already operate with large chains. We already support some big chains. We couldn't do that unless we had the level of diligence and stability in our platform.
 
But, really, that kinda, applies across the board to all of our customers, large or small.
 
So if there's someone listening right now that, like, managing the IT infrastructure for a large chain, do we publish this information? Do they need to come to us to find the documentation that we have? Or, like, how do they find, figure out what our infrastructure is and how it works?
 
That's one of the things that actually attracted me to ҹɫֱ²¥ is like, the openness of what we do. So we actually have platform documentation, which is completely open to anybody. You can go to the site. I don't know if you'll have show notes, but we can maybe put it on the
 
mews.com. Click above. Click down below.
 
Wherever it is. Like it's yeah. It's all completely open. So you can look at how we scale, how our database is available, what technologies we use, what languages we use.
 
All of this is completely open. We try and be as open and transparent as possible. We write a ton of blog posts as well. So we actually have, like, a tech blog where you can go and read about our practices, how we scale, how software is developed at ҹɫֱ²¥.
 
So we try and be as kind and open as possible. But, yeah, all of that is essentially kind of open for everybody. I think, like, maybe the question is, like, what would the expectations be for enterprise-ready? What uptime.
 
But you often see things like, SSO for login. Yeah. So single sign in, you know, role-based access, audit logging, robust APIs.
 
All of these things are things you often associate with kind of enterprise-grade, and we provide all of those things. Right? I actually think we go beyond it. So, you know, recently, we launched Passkey, right, as a way to authenticate and to log in.
 
This is like the future of how you log in. It's a very, very secure way to do that. Very few companies are actually doing that. So I think we're even surpassing some of the enterprise companies.
 
I know you know, Matt, that I worked at Salesforce before that. And even some of the practices that we do at ҹɫֱ²¥ are well beyond some of those kinds of enterprise software, companies, and we just continue to kinda push the bar on them. We can continue to launch new features that I think do sit in that kind of enterprise-ready class.
 
Because what shifted last year? Last year, something happened across hospitality that really shifted our focus towards really doubling down on security and talking about it publicly and developing features. But what happened last year?
 
We've seen a huge vector of attacks across the industry. Right? So we've seen these hackers try and get into systems across the whole of the hospitality industry. And the reason that we know that it's across the industry is because ҹɫֱ²¥ is a part of the retail and hospitality, cybersecurity, forum, and that's a group of companies that come together to essentially share their experiences, what they're seeing, and if it's a common pattern.

Chapter
The danger of phishing attacks in hospitality
 
The main thing we're really seeing is phishing.
 
Right? And so that's where these hackers, attackers, are trying to convince people to give them their credentials.
 
And the way that they're doing that, I think, is quite interesting. You posted something on LinkedIn, recently around this. What they're doing is they buy ads on Google so that if you are one of our customers, they'll go to Google, they'll type ҹɫֱ²¥ login, and the top result is a sponsored ad. It looks completely legit, but it's not.
 
Right? Essentially, what the hackers have done is pay money to Google to get to the very top. Then, when you click through to that site, it looks like ҹɫֱ²¥. Right?
 
It looks like our login page. And so what we're seeing is our customers are being tricked. They enter their username and login, but the site's not mews.com. It looks like it is, but it's like mewas.com or something similar.
 
And at that point, they've given their credentials over. Right? So they give their credentials inadvertently to the attackers. The attackers can then get into ҹɫֱ²¥ using those credentials, and then they'll scrape their data or reservations or things like that. But, again, this is across the board. We've had over four hundred fake domains.
 
So fake. But don't we have two-factor authentication deployed to all our customers?
 
How are they still, like, getting in even if we have two-factor deployed?
 
It's a constant game of cat and mouse in cybersecurity.
 
Like, you make one move, they make another. Like, they are dedicated and they are heavily backed financially.
 
You know, a lot of these are kind of state-sponsored attacks.
 
So in two-factor, yeah, you're right. You think, well, you've got two-factor. How could you do that? Well, they just pivot. So when they ask for your username and password, and, essentially, what they do is they take your credentials, send them to ҹɫֱ²¥, which triggers us sending you a code to your email or your device or whatever. And then they have a second page, which says, now give me the code.
 
And just like that, the person still thinks they're on the site, they enter the code, and then the attacker puts the code in. So they're even trying to get around kind of two-factor, which comes back to the point on passkeys.
 
With passkeys, that doesn't work.
 
Can you explain what a passkey is? Because I think a lot of people, I heard it for the first time. I was like, oh, sure. I have to fully buy a device. I have to plug into my laptop. Because I thought it was one of those, like, hardware devices I had to purchase, and I was like, I'm never gonna do that. But, actually, passkey isn't that.
 
No. That's right. So, passkey is like think of it like your mobile phone. Right?
 
So when you often log in, you don't log in typically now to your mobile phone. You use your face ID or a fingerprint. So it's exactly that. Like, you can use your trusted device to authenticate, and you can use any, you can actually buy one of those keys, like a security key or whatever it might be, or you can just use your mobile phone to actually authenticate.
 
So it's much more secure because I've literally got my phone to authenticate. It's a one-time setup. It's super, super quick. It's so easy to set up within ҹɫֱ²¥.
 
And from then on, you can just use, like, a password manager or a device, and it's so frictionless. So it's not only frictionless, but it's, like, very, very secure to do.
 
Actually, so what's better?
 
Like, passkey or single sign-on?
 
Both are, they're equally important, so we have had zero users phished who use SSO. So both are great options.
 
But I would imagine if you're a smaller independent hotel, you're probably not gonna have an entire infrastructure to set up single sign-on and manage all of your systems. So if I think a small hotel would probably go for passkey, whereas, a mid market, like, a hotel group, they would possibly have single sign-on. What would you say to come because there are still a lot of hotel groups that work with ҹɫֱ²¥ that do not have single sign-on set up. So how would you convince them that that is the right way forward?
 
What's your sales pitch?
 
Yeah. No. Like, I think the fact that we've seen zero account takeovers at all using SSO. It is such a secure way to do it, and it's simple to set up.
 
It's so easy to set up. It's a one-time setup. And then from then on, you don't have to worry about it. You're using a single set of credentials that your team already use.
 
It's frictionless, you know, much like that.
 
And I think the thing that I love the most about it and, you know, we use it at ҹɫֱ²¥ for our systems as well, that when an employee leaves, and in hotels, we've got quite a lot of turnover, you don't have to log in to every system to remove that user one by one because you will forget one day.
 
And then that means that user has access to it. But with single sign-on, you remove them from the single sign-on kind of infrastructure, and it gets removed from all of the systems simultaneously. And because of the high turnover in hotels, I would strongly recommend anyone who's considering it to prioritize this over anything else to do today.
 
Hundred percent. That's a good point. Yeah. I think, like, it's centrally managed. So if somebody offboards or leaves, you don't have to remember, oh, now I need to go into ҹɫֱ²¥ and remove them from that account. They're already offboarded once.

Chapter
Future investments and developments at ҹɫֱ²¥
 
So what are you investing in the next twelve months?
 
A lot.
 
You keep asking for money.
 
Yeah.
 
We have, we actually have 2027 plans. We've got, like, a two-year vision for where we want to be. And there's a huge amount that we're investing in. As I said, like, we're constantly focused on performance.
 
We actually think that there's a huge amount of improvements we can make on the performance of the product, which will dramatically improve the user experience. So if you think about, like, checking in, which, you know, front of the house will do every day, those types of things need to be super, super rapid. So we're investing heavily in reducing those times to improve that user experience. So that's a massive one. Another one that we're looking at is central identity.
 
So as ҹɫֱ²¥ has increased its ecosystem, you know, we have RMS now. We have, events management system. We are broadening the ecosystem.
 
We want to have a single login for that. So once you're in ҹɫֱ²¥, you're into the entire ecosystem. That's something that we're probably looking at late this year, early next year.
 
But it sounds easy. But it's like, I'm like, great. Just use the same login for everything. But, actually, there's a lot of complexity behind the scenes to get that deployed, I'm assuming.
 
Exactly. You need, like, an identity provider, and there's there is quite a bit of complexityin there to make sure that it's kind of consistent. But, again, similar to the SSO thing, it also means you're managing just one account. So it should, like, be more frictionless for customers and improve their experience of just managing one account versus one for each of the kind of products in the ecosystem.
 
Nice.
 
We're investing massively still, of course, in security. So we actually have, like, quite a large security team today, and they do both their preventative things. So for example, every line of code that we a developer writes at ҹɫֱ²¥ is security scanned. Like, literally nothing gets to production unless it's security scanned for potential vulnerabilities.
 
But they also do things like remediation. Right? So not just the prevention, but if something happens, how do they respond to that security incident? We're actually splitting those functions out. So we're gonna have separate and dedicated teams for kind of prevention and remediation. So massive, massive investment in security. And more broadly, we're hugely investing in the infrastructure.
 
Today, we have like a large monolithic system.
 
Again, we deploy it twice a day. We are now building out, we're breaking the system down into much smaller parts so each team can build and run their systems, like, and deploy multiple, multiple times a day. We will get to the point where we are releasing hundreds of times a day. Today, we're at two. So there's a heavy investment in that.
 
Nice. My last question. If anyone's been listening this far to our conversation, what's the one thing you want a hotelier to walk away with that makes them sleep better at night?
 
I mean, I think the fact is that we have over a hundred people just looking at this. Right? Like, this is not like a three or four-person team. We have over a hundred people on the platform, and every line of code that we ship, it has the guest in mind.
 
Right? Like, we obsess over reliability, security, performance. It's a thing that's never done. Right? We constantly obsess about kind of, improving.
 
So whether that's, like, check-in, payments, reports, like, the entire platform team is working 24/7 to make sure that everything kind of runs smoothly. And I don't think people often see that. They just think, oh, there'll be a couple of people running the system. Yeah. But there aren't.
 
On these bicycles keeping the system, like, electricity going.
 
But we are constantly not only investing in kinda keeping up, but we have multiple teams, like, driving the future. And the next twelve months are gonna be super exciting. I think you're gonna see a massive change in the performance of the product, stability and everything is just exponential, I think, in the next twelve months.
 
Nice. Thank you so much for joining me today. Like, it's a topic that is really hard, but I think you made it digestible. And I'm hoping that that's the response we'll get from hoteliers. But I really appreciate you showing up, and I know that it's not always the most comfortable thing for someone from the development team to step into the limelight and to talk about these really complex things. But for me, it made sense.
 
Appreciate it. Thank you for having me.
 
Thank you. Bye.
 
Bye.