We were able to have a pit tour, see some of the IT infrastructure in place, and learn about the role the CIO plays in a modern F1 racing team, with some very interesting observations and lessons to be learned.
So, below you'll find the audio interview I conducted with Hackland, as video interviews are not permitted at the event due to rights for video being paid for by the broadcast partners. But it was a great conversation nonetheless, captured in full and also ably transcribed by the rather cool Otter.ai app.
So, without further ado, the interview is below, and an accurate transcription of the entire interview is below the video!
Here is the interview transcript:
Alex Zaharov-Reutt (AZR) 0:00
Hello. And thank you for joining me for another Alex on Tech and iTWire interview. I'm here today with Graeme Hackland. He is the CIO of the Williams F1 team. Welcome to the program!
Graeme Hackland (GH) 0:10
Thank you very much. Hello.
Now we're on audio today because of F1 regulations that prohibit filming. Can you please tell us a little bit about your history in the world of tech and how you became the CIO of one of the world's great racing teams?
Yeah, so I came into Formula One in 1997 and it's over 20 years ago. Yeah, quite quite a while ago there and things were pretty different back then, actually. And I just I got lucky I never thought someone like me gets to work in Formula One right? And I remember saying to my wife - I'll do it for a couple of years and then go get a proper job and actually it is a proper job. I love the the constant change, the challenge. What we do is get two cars around a track but actually there's so much more to it. Yeah, I just love the environment.
So what does a day in the life of an F1 CIO look like in between races, on race day, and in the offseason? That's three questions in one but three different time periods for you.
I'll answer the easy one first. On race day, nothing and no-one lets the CIO near anything on race day.
It's the mechanics' domain, the drivers domain...
I have five operational IT people who are, you know, making sure that everything is running is working by race day, everything is stable. We tend to set up on a Tuesday, sometimes on a flyaway like this - we call races so far away flyaways we'll actually start setting up on a Monday, so come race day, everything is stable and running and they really don't ever need me. If if IT were to cause the car to stop then then I would be involved.
That's never happened?
Absolutely. In the six - this is my sixth season with Williams - and so far has not happened.
So what about in the offseason or in between races?
Yeah, so in in the offseason, it's all about getting the new car ready. So our focus is on making sure that the wind tunnel computational fluid dynamics, the virtual wind tunnel, the design team, manufacturing, they all have the systems that they need, and will also be refreshing all of the race team equipment, we've got two racks of equipment that we travel around the world.
And we'll have made sure that that's all refreshed and serviced and hardware gets replaced every two years to make sure that we get the reliability we need. So yeah, there's a lot of work done in in the winter in the offseason.
And then between races, it's, you know, you're asking specifically about the CIO role, really, my role is about managing risk. A lot of the time, it's about making sure that we protect our IP, and we protect ourselves and other people's IP isn't coming into our network. And, you know, viruses and malware and all of the threats out there, ransomware, that we're protected against that. So my focus for a lot of the year is around is around protecting the team.
So can you go into a little bit more detail, I mean, you took us on a bit of a pit tour before this interview, and we were able to see some of the servers and all the technicians and all these walls of screens. But some more on the IT that is in play in 2019 for an F1 team? And I'm assuming that, as you mentioned before, most of this stuff has to travel with you around the world, you got to take it with you and set it up.
We bring the minimum we have to, to the Grand Prix circuit, partly because we have to pay for transporting it. But also we're trying to leverage cloud as much as possible, right. So I can't bring to the track as much compute capacity as we need, I can't afford to, whereas I can afford to flex up in the cloud for the 4, 5, 6 days that we need it.
So the cloud has been really useful for us in terms of that. So we'll be doing computation in the cloud as the cars going around the track all weekend. And in the past, we were somewhat limited. So jobs will take longer sometimes between the first practice session and the second practice session, you wouldn't get all the results. And so you might not make some changes to the car that that you would nowadays.
When we are actually able to put the right data into the hands of the right engineer in near real-time so they can make decisions that affect the next session. So a lot of focus has got into that making sure that they really can make real-time decisions.
And what sort of connection do you have to the cloud? How fast is your link?
So we partnered the season with Tata Communications and they're providing us with 100 meg MPLS link wherever we are in the world.
What's an Mpls? People know Mbps is megabytes per second.
What's Mpls? That's a very good question. I do remember the term but it's a secure link, right? [Editor: It's Multiprotocol label switching]
No-one can get into the link. So we're working at 100 Mbps.
So as the cars going around the track, within point three of the second of the car generating data, we will have it back in the UK for the engineers who are there to work on, which is, you know, when I termed near real time.
This is one of the furthest tracks we come to, obviously, so the latency is longest from here. Yeah, point three of a second, the engineers and back at base can still talk on the intercom, can look at video data, the photographer's data that we have and obviously, the car data in near real-time, which is brilliant.
So just out of curiosity, what part of the tech in the F1 cars does the CIO have responsibility for and/or oversight over?
I don't know whether I should be really pleased to tell you this or not, right? Nothing! Right. So the onboard systems are done by our control systems groups, and electronics and so on. And it's really only once the data comes off the car that it becomes IT's responsibility and all of the infrastructure to support that car. If you want to download any data to the car, you need to do it with a laptop. So we support all of that. But the onboard systems thankfully, or you know, we're not part of that.
Now, obviously, the protection of all of your data is paramount. You did already mention a little bit about it. But I'm guessing you're dealing with petabytes of data, certainly gigabytes and terabytes. But there is so much data that's coming from these cars. And you're managing networks, you're learning about all that data from race to race and year to year, and you're fighting cyber espionage as well. But what else can you tell us about this at aspect of all the data protection you need to have in 2019?
Yeah, you're right. I mean, we do have terabytes of data. And we are approaching petabytes. And actually, a small proportion of that is the actual car telemetry. That's not by any means the biggest amount of data that we generate as a team. At the track, it's video data for analytics back at base. I think by far and away aerodynamics data, either from the wind tunnel or from computational fluid dynamics (CFD) is by far and away the biggest data. And that's where we get into the terabyte. Because if you think about it, Formula 1 is just about two cars around the track.
So yeah, but we still generate, you're right, a right huge, huge amount of data. So the kind of risks that we face are the loss of IP that's valuable, and trusted insiders who have access to their data.
We at Williams also have an advanced engineering company that's commercialising what we do in Formula One, lightweight materials, electric vehicles and so on. So that data is very valuable for a long period of time.
Formula One day to three years, four years, maybe the data is valuable after that, no one cares, no one's looking at 2013 car data, the cars are so different now. And come 2021, because again, it will be a whole new generation - completely different.
And it keeps everything fresh and exciting.
It makes it great for the engineers and the aerodynamicists and the challenges are constantly changing. So that's a good thing for us. But it means that that IP, especially in advanced engineering division, that IP is very valuable for a long period of time. We're putting it into road cars, you know, that's a different challenge. And protecting that data is a different challenge to F1.
So we focus a lot on how do we protect the infrastructure.
So things like ransomware, as you mentioned earlier.
And many years ago we have had two ransomware incidents. Before we partnered with Symantec and Acronis on our security software. We were hit by two ransomware attacks. And yeah, that's not comfortable, we're not prepared to pay the ransom so you run the risk of losing data and I guess denial of service type attacks is you know, is a potential problem that we that we worry about. The car, you can only make changes to if you plug a laptop into it - you can't make changes over the wire so that's a good thing.
We're applying that knowledge that we had into road cars and some of the road car projects that we work on.
So one of your partners is Acronis and the question is, how is their technology helping you do a better job and how does it augment the security of your operations? What can you do with Acronis now that you couldn't do before they were your partners?
So there are a number of things that that we started to struggle with, in terms of backup window, right, we all thought that we'd solved the backup window issue.
And then the data got so large that the backup window became a problem again. So there was systems that we are not able to complete the backups in good time.
So I couldn't guarantee to the race engineers or at an emphasis that we that we could restore their data, because obviously, backup is just half the story as well, we may not be able to get to data in a point in time for them.
So we started hitting backup window issues, which was a real problem. And obviously, the ransomware that I mentioned, we're really concerned that the ransomware, if we got hit again, would get to our backups and an attack that data.
Also the third area, we weren't able to back up any of our cloud solutions before. And so working with Acronis gave us the confidence to be much more aggressive with our cloud strategy. In the past, we would only work with kind of tier one vendors who we knew where doing the backups themselves and so on. Now we have much more confidence to go with whoever we want in the cloud. Because if backing up all of that data and something happened to the vendor, we would still have the data.
So what's one of the most challenging IT experiences you've faced, as Williams CIO that you're able to talk about? I guess ransomware is one of those...
So the most challenging thing is when we lose data, and one of those ransomware attacks did actually result in us losing some data that we were not able to recover. That's what I hate the most, I find that unacceptable, that we would put the team in that position. So we focused a lot of making sure that that that can't happen again. But there are challenges every day around protecting the team, our reputation, and our IP and our customers in advanced engineering, making sure that we protect their intellectual property as well. So those are the, I think, the biggest challenges.
And so what has been the most surprising part of the job thus far, in a good way?
Yeah, well I've been in Formula One for a very long time, and it doesn't mean you don't get challenged. So there are things that we're looking at for future technologies that I think is the most interesting. So when we start to look at AI, and the role that that could play in Formula One. So could we call a pit stop without any human intervention in the decision? At the moment, no, because I don't think the humans will let us, but the technology, yeah, we're starting to explore that technology, and it's evolving, maturing, and I think we'll get there very soon. I predicted that by 2020 we'll be able to do it. It scares me now because that's next season!
Five years ago, I predicted that we would be able to do it by 2020, I think the technology is getting mature enough that that we probably would be able to do it by next season.
And there's more sensors in the car that can feed the AI engine within enough information, that it can then confidently make that decision whereas you didn't have those sensors before, they weren't as advanced as they are now, is that right?
I think you're picking up on exactly the right point. It's your to confidence in the data, right? So if the data is inaccurate, the AI engine is going to make the wrong decision.
Garbage in, Garbage out...
Where a human might think about it, because they've got experience. And they might think, well, this doesn't feel like the right decision, and they might question it, and then you may miss an opportunity, and think damn, we made the wrong decision. But some element of thought went into it. And that's where I think AI will be good because it will learn from from history, and hopefully, you'll make the right decision more frequently.
So you're right, making sure that the data is not bad, is going to be really important. So that's where partners like Acronis are going to be really important to us.
How do we make sure that our data is not tampered with or corrupted, and then we make the wrong decision. We've seen humans make the wrong decisions based on GPS data, for example. So positioning of the cars around the track is sometimes wrong at some of the Grand Prix. And you might call your driver in, thinking now's the time to pit but actually - and we've seen a couple of instances that where the GPS data was out by six seconds, 10 seconds. And that's enough for you to lose a place on the grid. So yeah, making sure that the data is not tampered with and not corrupted in any way or data is not missing. And so the AI engine doesn't make decisions on bad data. But that's just as important for the human. So I think initially we'll augment the humans and make sure that they have better data to make their decision. And then ultimately, we'll be able to give them more interesting things today.
Now, I always finished with three questions. The third last one is always about the future. So you know, you've talked about what you might think might happen in 2020 with AI, but what about, say, 2029? How do you think F1 will evolve over the next decade? Just look into the crystal ball.
Yeah, the great thing about Formula One is it constantly evolves, right, it's ... every... no two years of the same. We are constantly making changes to the car technology itself. Those those cars are incredible right, a V6 turbo with two energy recovery systems setting the fastest ever times in Formula One history. That's amazing. And I think we'll just see more and more of that as the cars evolve, and as society evolves, will see that in Formula One, too.
And that car you talked about with the recovery systems, is that Williams cars or whose cars are that...
So that's the grid. So the whole Formula One grid is running these V6 turbos with two energy recovery system. So one takes energy from breaking and one takes energy from heat from the engine and converts it into power that the driver can use.
And when you think F1 and FE will merge? When will we see less of the petrol?
We already are, right, so we're using significantly less fuel now than we ever have in the past - about a third less fuel than we used to in the in the V8 era. So I think we are converging. It's a hybrid car right now. Will we go full electric? You said, what, what did you say? 2029?
Yeah, well, 10 years from now.
Yeah, 10 years. That might be too soon. Because I think there's a place for both Formulas. We're involved in the Formula E we're involved in the in the Formula E electric series. think there's a place for both.
So my second last question is, what's the best piece of advice that you've ever received in life to help you get where you are today?
So I've had three or four mentors who have really helped me all through my career. And probably the most important advice I've received over time is about people, surround yourself with people who have complementary skills to you and who are not people are just going to say yes to you, who are actually going to challenge and the management team that I've built in Williams, their job is to make sure that we don't miss anything. So it's not just to say yes to what I recommend, or for the rest of us, just to say yes to what they say. The way Formula One works is in a very agile way, right?
It's like an agile development team!
It is, right? They put something on the car they tested, if it works, it stays on. If it doesn't, it comes off and they replace it with something else. And we're doing the same thing in IT. We want to make sure that we challenge ourselves. So yeah, I think one of the best piece of advice I got was surround yourself with difficult people who are going to challenge you.
And my final question is, what's your final message to iTWire viewers and readers and to your current and future partners and F1 fans? Bit of a catch-all.
Yeah, I think this is one of the most exciting times to be in, in IT. I started in 1990 and I've seen amazing growth. I mean, in 97 in Formula One, all the data from both cars fitted on a floppy disk, 1.4 meg!
So data growth for everyone is just exploding. What we've got to be able to do is turn that data into actionable intelligence knowledge. And I think that's the challenge for all of us. Every CIO, I talked to has same challenges that I've got so much data, I've created a data lake, but now I've just got more data in a data lake. And so we're all focused on how do we get data, the right data, again, get mined into the data, turn it into actionable intelligence, and that's, that's where the pursuit of things like AI or blockchain, all of that comes from. It all comes from being able to make the right decision - quicker than your competitors, no matter what business you're in. The good thing about Formula One is every two weeks, we know whether we've made the right decisions or not.
For some organisations, you got to wait till the end of the quarter, but for us, our competitors are in the garages next to us. And we find that very quickly whether we've done a good job or not.
Well Graeme, thank you so much for taking the time and best of luck with this season!
Thank you very much. Thank you.