>>Alright I’ll hand it over to
these wonderful people. Give
them a big round of applause. [applause]>>Alright ah ok so
you guys are all here, I’m sure
you- you know what this talk is. You should pat yourself on the
back because this is the right
choice for the talk to attend right now. It is the best talk,
you made the right decision.
You’re life is already done, you’re good! You’re good for
life because uh we’re talking
about Cisco routers and secure boot uh and the title of this
talk is 100 Seconds of Solitude.
Uh my name is Ang Cui.>>I’m Jatin Kataria.>>Is that on?
>>No.>>Is this thing on?>>Uhh.
>>Hey uh my name is Jatin Kataria.>>Very good ok. And
what is primary main objective?
[laughter] Primary main objective is not Chinpokomon
Camp [laughter] it is Cisco
trust anchor. Uh and you know so this talk is about the work that
we’ve done over the last 3
years, uh around reverse engineering not just the Cisco
router uh in front of you but uh
a fundamental piece of hardware that underpins a lot of the- the
secure boot process of uh dozens
of types of Cisco devices ranging from you know small
switches all the way up to the
very large enterprise level you know core routers. So uh the
trust anchor is something that
we had no idea it existed when we started this project. And
we’ll take you through all of
that story but um ultimately what this revolved around was
the work that we developed,
techniques that we developed, uh to manipulate FPGA bit streams
in order for it to do something
useful. Uh so uh you know even though we’re gonna talk about
the security implications of uh
our techniques on this router, uh we’re really here to talk
well ok so the deeper primary
main objective is uh FPGA bitstream manipulation. And the
reason why this is is because
you know yes uh Cisco routers and switches you know all use or
a lot of them use this type of
uh you know Spartan’s Xilinx’s FPGA but so do a whole lot of
other things in the world. Like
you know uh advanced driver assistance cars, uh I’m sure
legacy weapons systems,
missiles, and all sorts of things and the uh you know the
techniques that we’re going to
talk about here are going to affect those devices too, so
even though this might seem like
a fairly large impactful vulnerability disclosure, uh the
real impact of using these
techniques uh in devices we haven’t even started to look at
uh I think is much larger. So uh
you know like I said we did this for 3 years uh and there is a
long story which I’m gonna take
you through over the next 235 slides. And yes I have exactly
those uh and but before we start
you know this is kind of an epic story you know there’s love and
loss and betrayal, more love,
and friendship and defeat and you know redemption and and more
friendship uh so we’re gonna go
through the cast. Uh Jatin, over here, right. That’s him. Uh Rick
is actually in the front row,
that’s Rick. Joey is sadly not here, I think he’s at a wedding.
Uh this is me, I’m wearing
sandals, just like the photo shows. James uh right there ok.
And um well over the last 3
years uh at Red Balloon Security, we’ve invested a ton
of our resources in creating
this massively complicated highly intelligent automated
testing framework uh abbreviated
to BRIAN. You know it’s so complicated that some might even
say that he is borderline
sentient. Which I think is ridiculous but you know, we’ll-
we’ll get to that. Alright so as
with uh all great epic tales, uh we start in mid story! Ok so you
know we started this work in
2016 uh in 2015 Jat and I worked on this little thing where we
got code execution inside the
onscreen display controllers of all sorts of you know probably
over a billion monitors. Uh and
then in 2017 you know Rick and I worked on this thing where we
built a EMP uh electromagnetic
pulse generator, in order to defeat secure boot in trust zone
on a Cisco phone. So you know
those projects were pretty simple, straightforward, you
know there’s really no thinking
involved, I mean very easy to do. Uh so in the middle of all
of that you know I say hey guys
like our lives are so easy you know like it’s going too well,
uh let’s punish ourselves for
like no reason. And then Jatin says->>What the F man? Why?
>>Grr Grr Grr [laughter] Say
what? Right so ultimately you know fast forward to 2019 here
we are talking about all of that
work. Uh and you know it resulted in this thing called
Thrangrycat. Right? And
technically you can’t pronounce it like Thrangrycat because it
is the first vulnerability named
after three unpronounceable emojis so you can only come
close to pronouncing it but not
really, you can’t really say it out loud. Uh thrangry fact 1: uh
the domain name for this
vulnerability is emojis ok and for uncool kids you can type it
out as thrangrycat it’s fine. Uh
thrangry fact 2, we you know have all sorts of information
about the nature of the
vulnerability, you know how we did it. And actually I will
tempt fate and try to tap out of
this thing to show you our website. So if you go to our
website uh just a few hours ago
you know we put the slides up finally and then we also have
all of the libraries that we’ll
be talking about and the tools that we’ve built um it’s all on
GitHub it’s right there on our
website. So if you like any of this stuff and you’re
interested, check it out, the
tools are actually super cool. Ok so we wrote about you know we
had this like 4 uh reach
rational- uh 4 serial- a set of 4 reasons why we decided to name
uh our vulnerability after
emojis uh and I’ll just give you 1. Rule- uh reason number 2 is
that emojis are indexical to the
digital age. Right and uh that’s just 1 of 4 reasons and thrangry
fact 2 point 1 and this made my
heart sing when I read it, right. I found it interesting
that the section that explains
the name of this vulnerability is way more thought-out and
verbose than any other sections.
It’s almost they had like a linguist on the team and the
emoji name was his prideful
contribution. True fact. True fact. Uh thank you if you’re out
there, you’re good. Um alright
so just the immediate impact of uh this vulnerability uh it
affected like I said probably s-
I think something like 180 uh types of->>133.>>133 Cisco
devices and here’s a list of
some of them. You know it goes to like this page and more on
this page and this page and this
page and this page and this page and this page I think there’s 1
more. Nope, 2 more. 1 more.
Alright? Alright cool. So it affects a whole lot of Cisco
routers. Alright let’s talk
about how we did it. Uh you know so.>>It all started when we
wanted to uh hack ASR 1001. But
it was end of life and uh Cisco started uh you know sending 100X
in 2013.>>Yeah so right Jatin’s
like let’s look at the 1001. And I was like aw, we can’t really
buy this thing anymore so
whatever I see this thing called the 1001X, uh whatever let’s
it’s probably the same you know.
>>Yeah but uh what uh doesn’t matter we don’t care but
documentation, who cares about
the X, you know? X is for chumps.>>It’s probably fine
right? So we looked at it, we
you know signed ourselves up to do this work and you know to be
fair, look the left is the 1001,
the right is the 1001X, looks exactly the same. Uh you know
the 1001X uh it was kind of
expensive it’s like 11,000 dollars per router but we bought
it anyway to d- help everyone
visualize what exactly that is->>It’s on sale, no?>>Huh?>>The
sev- it was on sale.>>Oh that’s
right. It’s on sale so you can get it for a little bit cheaper.
Uh anyway so what is that? Let’s
talk about what that means so 1 dollar is uh 6 point 1 4 inches
by 2 point 6 1 inches. It’s
approximately 1 gram so the cost of that 1001X is approximately 3
and 3 point 5 8 feet or 22
pounds which uh to put in a unit that you guys can understand,
it’s basically 2 healthy 3 month
old babies stacked on top of each other. Like that’s the
right height and weight, that’s
exactly what it is. Ok. So the uh 1001X the X uh has this thing
called trust anchor in secure
boot which again at the time we had no idea what it is, uh so we
thought hey maybe this is just
like a new boot loader, whatever let’s, we can do it. Alright so
goal is uh what do we want to
do?>>The goal is to modify the ROMMon like we just wanted to
run our own firmware, our own
stack on the ASR 1001. It was- that’s what we wanted so uh we
have done in our previously work
and we thought you know like it will be easy, it was just like
change the root key, change the
firmware, doesn’t matter let’s just do that.>>Alright so and
uh this is kind of like iron
chef. The secret ingredient is we- we actually have 1 of these
things in front of you and if I
don’t knock it over the table, we’ll see what’s inside. Alright
if we don’t keep the lid on uh
it’ll the thermals are horrible so it’s not great. But anyways
so when you open this box uh you
see this thing over here. Uh it might look a little weird but a
lot of this is basically a
standard you know Intel based computer. Alright so the things
that we care about is um you
know obviously there’s the Intel Xeon processor so probably a lot
of that code runs there. Uh and
uh there is a Xilinx FPGA so who knows maybe that’s doing
something interesting. And this
thing has a number of different little FP uh SPI flash chips
that uh we thought might
actually be you know useful to look at. Uh and the rest of it
is a whole lot of complicated
hardware that actually makes you know the routing and switching
stuff work. But you know we
opened this box and we said ok you know we probably understand
something about this uh piece of
hardware, and you know this is somewhat like a normal pro-
computer but it certainly has a
whole lot of weird stuff that we haven’t seen before.>>Ok so
let’s look at the software now.
Let’s just. Um so eh uh if you guys ever worked with Cisco they
used to have uh this uh
proprietary boot console called ROMman so which allows you to
boot through TFTP, you know
change images. Uh so what they did was they changed to UEFI in
2013 and uh but they still
wanted to work with their ROMman, so they implemented a
PRE-ROMman as a Pre UEFI module.
What it does is that it manages ROMman. It also with those 2 SPI
flashes as you saw, as you can
see. It also handles the updates so in the Linux kernel if they
want to apply a patch to the
bootloader the Linux kernel uh kernel will update one of the
SPI flashes and then the
PRE-ROMman will validate uh that whether it is right or not and
if it is not, then copy the code
in copy again. Uh pretty common stuff. Uh then they had Rom- uh
then they had ROMman and uh
basically this is implemented as a DXE module uh again it does
exactly the same thing it can
allow you to boot uh d- Cisco images um through DFTP it also
allows it also validates the
next uh operating system it’s gonna boot. And the boot loaders
um they are OS stage. And uh it
also has a reverse module which allows you to uh in- introspect
what is happening in the DXE.
>>Oh by the way so ROM doesn’t stand uh for what you think it
does. It’s not actually Read
Only Memory, it stands for Romantic Monitor. [laughter]
True fact.True fact.>>So uh
then comes the Linux kernel, uh so they booted uh they started
using uh you know uh- uh off the
shelf Linux kernel on which they booted their own Cisco stack
running uh as a privileged
process. They also wrote a process manager which uh manages
this process. So anything any
kind of crash happens in the IOS it reboots the whole, uh system.
And once you combine this, this
becomes your IOS XE stack and if you go to their website this is
the most secure stack. Uh so we
while doing this analysis looking at everything we s- s-
saw no hashes on the UEFI no
certs on the UEFI so it should be like really easy mod because
our goal is just to run our own
stack. And uh we also disabled uh some of the checks which were
in the Pre-ROMMon and we mar- uh
booted the modified firmware and everything worked. And it was,
this is pretty simple. Uh so but
wait, then router reset. Then we saw like wow it reset after
every 100 seconds which doesn’t’
make sense. And uh we spent a lot of time. And then I->>Well
so first it was uh you know it-
it you know it resets- well he was able to get code execution
on the Xeon processor and it
allowed us to do it, and then you would eventually kind of
reboot and then uh at first you
know he was like “Well, it reset once in a while, maybe a
minute.” Right?>>Yeah.>>And
then he was like “Eh it’s kind of like 100 seconds.” And I said
well, how many seconds? Like
exactly 100 seconds? Because computers don’t like to count in
decimal you know, it’s not like
they have 10 fingers and 10 toes. So 100 exact seconds-
>>Yeah.>>Was kind of a hint to
us that there’s something else uh that’s kind of like a you
know a piece of code that some
engineer wrote that enforces that 100 second thing.>>So this
is signified by a loud fan
noise. These fans as you can see 4 fans, make a very loud noise
and this was crucial in doing
all of the reverse engineering. This is probably the key, the
fan noise. So we came up with
the hypothesis of like what is this 100 second reset causing?
You know there are multiple X86-
because the route processor is in X86 code so maybe there are
multiple mitigations like the 1
is the you know maybe it is doing some kind of virtual
machine introspection. Uh but we
analyzed the code it was already disabled. Uh X86 has dozens of
watchdog timers. So we disabled
all of them because we had 100 seconds to run whatever code we
want. And uh that also did not
help uh then we went to SMI_EN and we saw like SMI_EN was
enabled so we disabled that. Uh
but it was still rebooting. So it was like um, I don’t know
man.>>Yeah so we we
hypothesized like is there like a magical deity in this
processor that somehow despite
the fact that we control every instruction that executes on the
processor, still it’s watching
us and tempting us right. It allows us to execute, like our
hopes are coming true, right?
And it waits for 100 seconds before it takes our dream away,
right? Like what kind of thing
is this? Rrgghh. Alright so you know normally eh every time we
asked this question like how do
we mind read the computer? We don’t understand! Uh we
typically elect to default back
to electromagnetic emanation. And if you look at some of the
past research we’ve- we’ve done
this is a really useful tool. Uh and a way to kind of figure out
you know what the computer is
kind of doing without really understanding a lot. So you know
basic stuff like when you push
an electron through a wire, right, it’ll like sit and do
some magnetic field, then an
electric field, so it kind of emanates right EN so when you
take nearfield probe to the
thing right you should be able to measure out the uh- you know
activity around certain parts of
the- the machine in order to kind of come out with a timeline
of what is probably executing.
So you know we have the CNC machine that we like to hook up
to the nearfield probe right to
do this type of you know analysis automatically. Uh I
think the zip tie broke so we’re
just holding it by hand but usually it’s automated. Uh we –
we swear right. So uh after
gathering some data here is what we have. Ok. So uh on this 1- on
the- all on the- way on the
right we have uh the emanation coming out of the- the CPU power
supply. Ok so there is a tiny
little dot uh where is my mouse up here right so this is uh you
know time right uh so you know
as we go downwards that’s later in time so little dots come up
right and a whole lot of uh
noise coming out of the power supply you know in like a
quarter second or something like
that. Now uh there’s nothing coming out of uh the CPU the
Xeon processor until basically
the bottom of- of- of this graph, so the processor is
sitting idle doing nothing but
the FPGA the Xeon Spartan 6 FPGA has this uh emanation profile so
you know a neat- slightly after
the initial power turn on right you see 2 little blips and then
it waits there for a while,
thinks about its life, and then uh later it starts doing a bunch
of stuff. Right and then the SPI
chip 1 of the SPI chips which is actually connected to the FPGA
right has similar uh
electromagnetic emanation profile uh in the beginning and
then you know it well it gets
red and then basically nothing happens so you know if you put
like this is like Blue’s Clues
right? You put all of these clues together I wonder what’s
happening? So power turns on,
right. The FPGA reads from the SPI uh flash uh its bit stream
right, it actually reads 2 bit
streams, and then it loads the configuration into the FPGA and
then the FPGA does some magical
thing right and then basically at the very bottom of this, this
is where the uh the- the Xeon
processor starts to do stuff. So uh you know we start
hypothesizing that this is
basically the mean little deity that’s hiding in the machine
that’s taking our dreams away
after 100 seconds.>>You took my joke. [laughter]>>Alright.
>>The deity 1.>>Aww.
>>[Chuckles] But uh there is a slide. [chuckles] So uh there
were a number it’s coming out of
this SPI uh and uh we saw you know like it was not available
in the boot loader and turned
out uh we had 100 seconds so we can dump this code. Uh and uh
these were interrupt handlers uh
for the real mode. Uh basically the BIOS/ROM/vBIOS which is
present in that range. Uh then
we also you know alaysed we saw some uh address ranges which was
like uh 0XFED4 as you can see.
This is usually used for external devices uh which are
map um, uh you know mme remapped
as uh IO devices uh for SQ’sI you know bus which is basically
a serial core interface. And uh
once you once this also gives us like kind of an idea like some
other IO device is mapped in and
X86 is uh- uh reading it’s state. So just to validate we
also hijacked the first X86
instruction, wrote the whole serial driver. Literally
validated and there was an
external entity which was doing it, and that was the joke, but-
>>Well wait wait hang on. So
this is where you know he put in this side and I said well you
know let’s not be culturally
insensitive so I added- we do have a cat vibe and then the
rest of this presentation so cat
buddha, cat buddha, cat buddha. [laughter] So there is a deity,
it is a vengeful 1.>>So uh we
wanted to uh you know we knew that FPGA is doing the reset uh
so this was our assumption at
this point. Like FPGA is booting, uh it’s validating the
UEFI so any bad change in the
UEFI uh s- address space uh you know causes the whole thing to
uh reset. And also Pre-ROMmon
checks ROMmon, ROMmon checks the LINUX OS um it’s kind of uh you
know secure boot flow. So then
we wanted to find the reset pin and uh uh it was the first
10,000 dollar device which we
did because it was a PGA chip and->>Well, so, and this is a
large part of the difficulty of
doing this work, because I mean seriously who can afford to just
blow away 10,000 dollars by like
touching a pin wrong you know. Th- that was for all of us uh
and we’ve destroyed many more
than just 1 of these. Uh so this was uh for all of us probably
the most expensive soldering
mistake we’ve made in life until this point so in in honor of
those mistakes we’ve built the
fail fort! Right? So the fort grows every time and there’s a
whole history mythology about
the rise and fall of fail fort but anyway fail fort version 1,
10,000 dollars, uh humble
beginnings.>>This is the start of the civilization. Um then uh
we wanted to test this theory
right uh and RTL reconstruction is really hard, and we’ll come
back to RTL- uh RTL. What is
RTL? And then uh but we still wanted it and uh uh I want just
like to pull the pre-reset pin
high and I’m a systems guy so I go to either Ang or to Rick for
my hardware problems and uh Rick
said you know this GI Joe guy over there like “I’m gonna take
a 10k ROM resistor and I’m gonna
just you know keep it high always.” And then it again
costed, it basically costed 1
dollar per 1 ohm. And it was total cost was 20 thousand
dollars.>>Alright so now the
router sitting there doing like completely unfazed except you
know the ones that have died, uh
and us we’re negative 20,000 dollars uh, in a- in a whole. So
version 2 you know we started
racing recklessly toward fail sky.>>This is- this is where we
uh you know took over the temple
that John built. Uh [laughs].>>Alright so you know at this
time ano- another 1 of our
colleagues, you know uh Joey was sitting and saying “Well, man.
Why are you guys like breaking
all of this equipment? You guys suck. Like why don’t you try to
Google this, you know?” And he
said like “I’m a level 9 Googler so look I type into the computer
thing and I found this patent.”
So here is a patent uh awarded to Cisco in 2012 that talked
about a lot of the things that
we were kind of seeing. Right so it is a patent on secure boot
through an external device and
it ex- specifically said you know you can implement this
using FPGA and it actually has a
very similar diagram to the thing that we came up with. So
anyway this document, super
useful. And 1 key thing that the document said is you must
implement the trust anchor which
is uh I think what they called it in this patent with an
immutable uh thing right because
if the attacker can m- like change it then the whole the
premise of it goes away. Uh and
they also said that you can implement it with an FPGA. But
you know we’ll get back to that.
>>Uh so wait- so this is honestly if you see it
disconnected through LPC bar, so
the way it is actually doing it is like it turns off these
upgrades so uh we had the code
hash to just reset.>>Alright so at this point you know we can
we’re kind of getting somewhere
but you know for- for us and I think for a whole lot of other
folks out there, you know we’re
firmware people you know maybe software people, maybe some
hardware people but all of us
found FPGAs to be scary. You know it is a mystery thing that
is really hard. Lot of smart
people working on it and uh you know they haven’t done the thing
that we needed so you know we
kinda you know said “uh this is so hard!” then we said “well you
know what, like why don’t we
stop? Let’s give up and say we didn’t.” And then like a year
goes by, right because uh you
know we have actual work and you know we already defeated twice
like 20,000 dollars in the hole
um and then you know 1 night Jatin was like->>Yeah yeah I
basically went to Ang and I said
like give me 20,000 dollar more.>>[laughs] Let’s do it! So->>I
can- I can hack FPGA.>>Yeah and
I said fine let’s go do it. So uh he read a ton of docs and
he’s gonna talk about it.
>>Yeah. So uh again as I said like I’m a systems guy so you
know how do you understand
FPGAs. And uh->>Wait actually, we we never defined what is an
FPGA? What is that for?>>Yeah.
So what is an FPGA? Right like it looks like a lot of blocks,
lot of registers, lot of flip
flops, I have no idea.>>Field programmable gate array.>>Oh I
see.>>Yeah.>>I still don’t
know that. Ok. [laughter] So uh how does the FPGA design flow
works, right? Like uh the coder
basically provides the hardware circuit, and then the SGA
hardware description language uh
the render dual chain synthesizes it, places it, maps
it, all this information about
the turntables about what is the memory initialization, routing,
and so forth. Uh gets encoded
into a bitstream which gets encoded into a binary which we
call it a configuration
bitstream. So any time FPGA gets configured how the uh you know
because it is an ICU which has
be- benefits of both hardware and software, uh so it has
hardware logic block to
implement to do competitions and s- um it can be reconfigured as
software. And there are
different types of FPGAs there’s SRAM based FPGA which is
basically boots uh it’s
configuration bitstream from an external static memory. Uh so
any time it reboots it has to
reload itself and that’s what we’re talking about here uh for
Cisco ASR. And then the other 1
is the flash based in which the flash is automatic uh is already
present on the die it’s less
power consumption, it uh boots really quickly. And then the
other 1 is the antifuse which
is- doesn’t allow you to configure again, so it’s not
kind of FPGA. So but still what
is FPGA?>>And now we learn->>Yeah>>About the advance of
FPGA because that’s how smart we
are.>>Yeah and beginning FPGA and I just learned Field
Programmable Gate Array. Ok.
>>And actually you still didn’t know what FPGA stands for. So
good.>>[laughing] So FPGA can
be thought of think of as like a combination of blocks you know.
Each block does something and
they basically correlate with each other. But again, what is
FPGA for systems people? Why is
it called to effects? So this is very important because what
matters is like if we can
control the Y and the X with irrespective of the logic F,
then we win. Doesn’t matter how
complex it is. So let’s go inside like what are the basic
blocks of the FPGA? IOB is a
group of basic elements which drives uh the output and the
input functions of the FPGA and
the value could be 0, 1, or Z. But really what is it? It is
just a thing which is connected
to IO pin and it gives 0 or 1. That’s all you care about. The
second thing is IO Interface.
This is how the pin gets the val- uh uh the value from the
rest of the logic and provides
value to the uh to the other part of the logic. And the third
is BRAM which is just memory. It
can be of different data bit uh you know it could be 1 bit, uh 1
width or 2 or 32 or it could be
uh for Spartan 6 it’s like 18 kilobyte 9 kilobyte blocks. You
can configure it you can make it
uh singular to a port. The other uh which does uh so complex
logic block is something which
implements the boolean function of the FPGA. There’s 2 slices, a
switch map matrix. Switch matrix
allows it to com- uh you know like- communicate with the rest-
uh rest of the FPGA or also
allows it to communicate it to other parts of the IO. Uh and uh
it- the slice contains flip
flops and LUT again, it doesn’t matter for a systems guy. Uh and
then there is you know slices
can be of multiple types, Slice X boolean function, Slice L with
the carry logic boolean function
and Slice M also allows- gives you some memory. Uh but this is
the complexity of the FPGA.
>>It’s so complicated. I’m I’m bored just->>I know!
>>listening to this. Right?
Because there’s so many you know like little nuances of piece of
ar-like all of the stuff and all
of the innards of this is uh all almost always in commercial
FPGAs kept as a secret. This is
a proprietary thing you know they allow you to they give you
the tool chain that generates
you know from the- the hardware description language, that
bitstream that configures their
specific vendor specific you know FPGA and it will work but
they don’t actually they will
generally never tell you the- any of the internals of how that
bitstream is read and what it-
what that bit actually does. But you know you have to know all of
this stuff about the components
and you kind of match up just depend on the fact that the
magical tool chain will give you
the bitstream that will magically configure this
hardware thing to do what you
kind of told it to do through you know a VHTL or a verilog.
>>So uh other resources, doesn’t
really matter. So what is developer domain, right? The
developer domain is like you
specify the source code and the attacker domain is the
bitstream. So how do you reverse
a bitstream?>>Well like before that let’s go back. Right so as
an attacker you know you don’t
have access to any of the information about the hardware
description language, you don’t
actually even know the- how the- the various blocks are routed.
All you have is this you know
end the result bitstream like literally a stream of bits that
is fed through the FPGA uh and
magically like I said you know that bitstream makes the FPGA do
what you think it should do. Uh
but as uh an attacker uh all you have is the bitstream so you
have to reverse engineer the
bitstream you know like you would reverse engineer a piece
of binary with the machine
instructions and then figure out you know what the logical
constructs is in that massive
bitstream. Without any documentation from the vendor.
And again you know each vendor
will have a different format. Right each type of uh class of
uh FPGA from the same form uh
vendor will have a different format so the rules are
literally changing on you uh
underneath your feet uh and that’s what we have.>>K. So um
so we wanted to reverse the FPGA
right and uh there is some people who have tried it. JBITS
was 1 thing which uh Xilinx
released in 1999 it allowed you to like change logic. Uh but
then this start uh stopped uh-
uh supporting it. Then Bil, which it requires a netlist, it
was released in 2012 and the guy
did say that it’s very, it’s like impossible to reverse
engineer. And then 2017 BITMAN
it also allows you to like move logic, relocate logic. Uh.>>So
by reversing you know we mean
you take the bitstream and then the goal for the most part is to
be able to re- recover the uh
hardware description language and underlying logic that went
into generating that bitstream.
And once you have that, and if you did that 100 percent
correctly, uh you might be able
to modify that logic to have it do something different than its
original design. And then you
have to run the whole synthesis tool right and regenerate the
bitstream, well, you know re-do
the optimization routing and then regenerate a bitstream and
if you did that 100 percent
correctly, again without knowing any of the rules about how those
bits are actually used uh for
that FPGA uh then you might have a hope of you know taking
something like the bitstream
that we found in the FPGA that we think is implementing the
trust anchor, and then
alternating its behavior in a way that it might allow us to
bypass you know the trust anchor
capability. But the chances you know people of doing this on
anything but a very simple FPGA
uh is very low. In fact I- I’m not actually aware of anyone who
has successfully done that in
higher path on something like a commercial uh you know like a c-
you know a real world commercial
uh application of an FPGA. Uh you- on some commercial you know
bitstream. So->>So let’s talk-
>>Chances of that working, very low.>>Yeah let’s talk about the
p- I/O Pin.>>Alright so like
Jatin mentioned you know on the FPGA you have uh complex logic
blocks alright so this is where
you know your hardware uh l- description language with the
logic gets uh configured. Then
you have pins. Right pins are literally those little metal
things that stick out from the
chip right that you wire to other metal things on other
chips. So you know we’re humans
we like to design things on you know uh few layers of PCB right
so the number of pins uh we want
to keep reasonable and the largest uh you know uh so-
Spartan 6 uh FPGA has something
like 560- 76 pins. Right? So you know that’s a lot but that’s a
very small finite number
compared to the logic blocks. Uh which go up to about 200,000 and
in in each block you have a
whole lot of uh you know like small- smaller logic blocks so
you know we have about no more
than 600 pins and we have 200,000 CLBs and you know
standard sort of uh approach to
FPGA reversing was to you know recover all of the logic from
all the CLBs and we looked at
that and we said well, you know, we’re not FPGA people. Like
we’re, you know, this is too
complicated! Right like this probably won’t work, a lot of
smart people have tried and
haven’t really gotten far, but why would, why don’t we look at
this? You know there are not
that many pins. So if we can figure out a way to s- uh- p-
deterministically and reliably
uh manipulate the pin, who cares about what the logic is doing?
You know if we can disconnect a
pin, right, and hook it up to something else that we supply,
then it’s 1 of those things that
like if the FPGA falls in the forest, like did it really
happen if no one hears it? No!
Right? Because uh you know if we take a pin, disconnect it from
its logic uh and that pin is
still connected to the rest of the circuit board, and then we
bypass that logic. Right so uh
and this is an actual plot of the Spartan 6 Family, the
largest m- no- most dense chip
uh has yeah 576 pins and 150,000 logic blocks. Right so if you
put that you know actually in
scale, the graph actually looks like something like this right?
So and this is not even to
scale. That curve grows really fast so 1 of our intuitions at
the beginning is, forget about
reconstructing- reconstructing the logic, uh, it’s too hard.
Let’s look at you know uh a way
for us to reliably manipulate the pin itself and see if we can
do that.>>So uh so uh you know
bitstream does come um optimizations and there are no
optimizations in the IOB. This
is uh 1 of the logic uh complexity we are looking at so
this is a shot of the 56 we
wrote and uh this is a very small part of that logic. It has
like thousands of CLEs and- but
what is interesting is this thing. Right like it this is the
binary presentation of the
bitstream. The top part represents the complex logic
block which is implementing the
boolean functions and the 7 er- yeah 6 of those are basically
the IOB blocks. Each IOB blocks
is 8 bytes in Spartan 6. And if you uh can- can control if you
change a little bit in this, you
know like a byte here, you can change the value of the logic,
uh of the IOB from 0 to 1 or 1
to 0. And you can ch- uh you can also change these 8 bytes
represents the characteristics
of this IOB. So you can convert it from uh uh you know like
input to output. By just
changing 1 parameter. The number of rounds in the shadow 56, look
at the logic changing but the
IOB remains the same. And this is the test. So the only diff
was actually in the complex
logic block, it was not there in the IOB. And this is very
important. So as I said->>Well
actually let- let’s back. So the thing that you’re looking at is
an actual graphical rendering of
the actual bitstream generated from this little mock example
that we created. Uh and if you
can go back 1 more slide, right yeah. So the IOB doesn’t change
at all. It’s in very predictable
places. We know where the pins are because they’re numbered you
know like 1 after another right
they are uh designed on the PC board and they don’t get to
change because the PC board is
uh you know in hardware and if we figured out like which 8 byte
construct um describes which
pin, then we should be able to very predictably and reliably uh
you know change the behavior of
that pin uh and we have a much more hope of doing that then
trying to do RTL reconstruction
on the logic block.>>K so as we said like uh we can change the
output from uh to uh you know we
can assign the value 0 without depending on the f- uh the
function F. Uh then similarly
output to 1 change output to input, uh and the same thing
goes with the input. So uh what
we want to say here is that like you know the vendor is not
vendor should not be dependent
on the obscurity of the FPGA because as we saw RTL
reconstruction is hard but not
changing IO. So how do you reverse a bitstream, right? We
see biststream as firmware and
it can go with unpack analyze modify repack. And y- y- it has
you can talk about like it has
encryption but you can probably do s- uh you know like uh side
channel analysis to figure out
the ES keys. You know you can do fault injection to change to
basically skip this check. And
then you can also do photon emission analysis. Uh so our f-
development board which we chose
to do all this research was Spartan 6 uh LX45D. Uh uh it’s
the model of the spun. And how
do you do on the unpack if you read the documentation you can
do the unpack. Uh which is
basically like find the sync word, figure out which uh kind,
what kind of um uh spartan it
is, uh download it’s uh register description language, parse it,
and uh find the registered FDRI
which specifies the hardware circuit for boolean functions.
So uh then we move to analyze.
There are multiple types of logic. Uh uh type 0 pa- uh
frames. The first 1 is the uh
complex logic block, frame which and also it contains IOI uh umm
you know like how to configure a
BRAM and all that. Type 1 is BRAM, the actual memory, what it
contains. Uh and the type 2 is
the IOB itself. And all of these, which is really
important, is all of these are
serially laid out. So if you figure out the range of- uh of
each of those then you can do
analysis and reversing to figure out which uh pin is represented
by which bits. And you know um
the main important thing here is like if you figure out the
device layout you can exactly
pinpoint which CLB is used which BRAM is used and which IOB is
used. And each you know these
are details which you can find on our uh GitHub. Uh and uh
major represents so it is uh 2
uh SRAM is basically a 2D uh RA. And uh each row has multiple
columns, each column represent 1
resource. So you know like a CLB a memory s- uh memory slice MCLB
will be 1 column, slice LCLB
will second column, BRAM will be third column and like that. And
similarly each, each column has
its own area of rows uh- which defines which flip flop is used
uh uh which mux was actually uh
encoded. So uh we- we actually analyzed the bitstream and
created this really cool uh
visualization. My team has really worked hard on this uh
tool. Uh as you can see it
specifies the resource utilization uh which column is
used you can see this is for uh
the Mojo board uh which uses Alex 9 it’s a really cool dev
board. And this represents the
ASR bitstream. So you- you can see like which uh uh block which
resource represents the PCIE,
which resource represents the LCLB, and so forth and so on.
>>So this tool, can you go back?
Is- is on GitHub right now. It’s amazingly cool. You should go
totally check it out because uh
you know you can all that stuff that Jatin said you can read
maybe 2,000 pages of Xilinx
documentation and try to figure it out, great. Or you can use
this tool which has already done
that. And not only that, it will show you like every- you can pin
point to any piece of the binary
and it will show you what that binary- what what that chunk of
binary represents. Is it an IO
pin? If so, which pin is it? Right? Is it uh like a CLB if so
like what is the raw data in
that CLB? Uh and for us you know we after reading the- the patent
knew that ok the trust anchor
does a whole lot of magical deity stuff right and then if it
doesn’t like what’s happening
after 100 seconds it will literally assert the reset pin
on the Xeon processor and
physically reset the processor. So that is how the mechanism we
think works so if we can find
out exactly which pin uh the FPGA is using uh for that we can
do our thing and disable that
pin and make sure the trust anchor literally cannot reset
that reset pin and then we win.
And uh the next thing is ok so because the pins are set in- in
serially, in serial order uh we
know exactly what part of that binary it ought to be so the
only thing that we need to know
now is which pin uh actually is used on the FPGA which basically
involves tracing you know the-
the PCB on this router from the CPU reset pin to the FPGA. Which
is very simple.>>Um we have
only 5 minutes left so I guess we’re gonna go to the demo but
uh this is how the encoding
works. If you know the range, figure out how- which pin
represent- which bits represent
which pin and um you know all this code you can easily read.
Uh.>>Yeah we have a paper
coming out and we- we have a lot of documentation on GitHub. All
the details are there. Check it
out.>>And uh 1 thing I would want to say is that modification
is really easy uh they have like
implemented this 22 bit CRC for single event upset. You know uh
and uh that is also reverse
engineered you guys can just check out the GitHub but we have
to go. But uh you can also
disable the CRC. I haven’t tried that but uh you can just disable
using this register, uh and then
I wanna sh- uh the demo basically contains uh we have a
bitstream which enables those 4
pins and we gonna just turn this pin off uh using the tool which
is kind of a present again. And
uh as Ang said, we basically figured out which is the reset-
which pin is controlling the
reset pin and you disable that. And really cool thing part about
this is, like er uh there is no
patch available. Once it is allowed, we disabeld the
emulation of the FPGA so that
the Linux core actually can not update the FPGA anymore. So
that’s pretty awesome. Uh and uh
ba- but- I also want to talk about uh Rick uh found really
cool way as Ang was saying like
there are 600 pins in this case there were 296 and you know our
uh automated extraction
bitstream tool who is not here right now uh so he re- he came
up with the cool idea of JTAG
SCANCHAIN uh as soon as the fan comes up uh he checks the state
of all of those 296 pins using
JTAG and we found 10 pins. And 1 of them was the 1. And you want
to->>Sure okay so this is where
BRIAN comes in. You know we after years of investing in this
uh we just couldn’t make it work
so we got an intern. His name is Brian and I said Brian you touch
all the pins, you find out where
this thing connects. And he did it! He’s great! Thank you Brian!
I mean without him this work
would have not been possible because you know we need to
trace through like the 6 uh
layers with a PCB in order to find you know which pin goes
from the reset pin to the FPGA.
Uh and then well you know he touched the wrong pins so fail
fort [laughter] grows alright
[applause] eh which is fine. I mean it was a very hard job so
uh but then you know we were
able to make some progress uh and we were able to find that
pin right? Uh and the last
question is well, this is cool but how are we gonna do this
remotely? You know if it’s not
remote, it’s not really interesting, uh it- because if
somebody already has root uh on
the router then ok you can change all sorts of stuff. So we
said well you know it’s 2019
finding a remote you know code execution vuln and then
privileged esc- escalation has
got to be really hard. I mean people have been looking at
Cisco routers for quite a long
time uh so man like do we- can we even do it? Uh->>Yeah so
they have a driver basically
that’s how they use it. And uh we also did a lot of fuzzing on
all these protocols. Uh while
this was going on James come in->>Uh well you know so we’re like
man this is gonna be really
hard. Who knows if we’re ever going to get this to work. And
then James who is sitting over
there said ok guys, I got root. [chuckles] It was great and
then->>LUA is easy. You know
like he found a command injection vulnerability CSRF
vulnerability on the Cisco
routers uh which are patched now.>>So let’s just point that
out. You know he found this kit
that’s remotely exploitable uh you know command injection in
2019 oh a on Cisco IOS. So you
know for those who say command injections are gone it’s like
you know it’s not true. Uh and
this was a very simple uh vulnerability to exploit so man
uh this was great. So if you
couple any kind of code injection on any part of the
attack surface of any Cisco IOS
uh you know platform or device that’s affected by this, you can
immediately use that escalated
privilege and then update the firmware uh and well update the-
the bitstream and then disable
the- the FPGA right the trust anchor. So you can take a remote
exploit, couple it with this and
make permanent modification of the hardware that is impossible
to undo uh using software,
again. So with that, let’s um- and we broke some more, uh and
then final cost: 50,000 dollars.
Uh the fort is very large uh but as a result we figured- finally
figured this out. So I’m gonna
let Jatin show the uh the 2 demos uh we’re gonna exploit the
Cisco router and then show how
this is done and then um we’re gonna also show the- the Mojo
board demo.>>Uh ok so um yeah.
K. So this is what I want to show right now this is booted it
takes like 10 minutes to boot so
I had to boot it.>>So this is the output of this ASR 1001X
right here right it’s running
the latest firmware, well 1 minus the latest firmware,
because Cisco did patch this.
>>So what I want to talk about here is that uh look at this
message. As soon as it boot, it
says initializing hardwa- hardware. And uh that says
system integrity status, this
status is coming from the FPGA. And the PRE-ROMmon which we
talked about earlier is reading
this status and printing it out. So now we’re gonna go ahead and
exploit it. F*ck. [chuckles]
[laughter]>>Did we connect the wire?>>We did.>>Yeah we did.
>>We connected to the wrong
port. Uh 1 second.>>No really?>>Yeah. I have- nevermind. Uh
d*mnit. Sorry guys.>>We meant
to do this. This is part of the demo.>>Okay well.>>Yay!>>Okay
it’s still going on. So what it
is doing right now is that it is doing James’ hack uh getting the
root and then we t- uh took a
driver which was cracked RKO, I don’t know why they wanted to
crack it but uh so we took- we
hijacked the driver, we understood the- reverse
engineered the CPLE driver,
understood how to emulate uh how the FPGA emulates the SPI uh
flash for the uh Linux uh OS and
then uh basically apply a depatch. You only require around
15 bytes of patch to disable the
complete uh update and also disable the reset. This is gonna
take some time, I wanna go ahead
and like do the module demo.>>So just so you- we’re- what
we’re watching is a full remote
exploit of Cisco IOS uh followed by a full modification of
bitstream bypassing trust anchor
and then permanently disabling any future updates to trust
anchor via the FPGA with this
bitstream uh you know modification. So that being said
right look at all- think about
all of the devices that’s affected, this thing has been in
deployment since 2013 uh this
vulnerability is in all those devices so if anyone finds any
code eh uh any kind of code
execution on any part of that- that attack surface, uh there’s
a chance that this is not only
has affected your router but is persistent uh to the point where
in order for you to get rid of
it you, you probably will literally have to desolder a
chip right from your router or
your switch in order to test even to see if you’ve been
exploited or not. Umm.>>Ok so
uh as you can see this is a Mojo board um and yeah can you put a
light on?>>Yeah.>>So this is a
Mojo board. You will, in a minute. And uh what matters is
like those 4 lights, right? And
uh those 4 lights are representing 1 on those pins and
we’re gonna go ahead, run our
tool, which I’m gonna run again.>>Well so right now you know the
lights indicate you know the
pieces of logic that we’ve put in. Very simple to understand.
You know like 1, 1 thing says ok
uh you know pull, like connect this to logic that always uh
makes the pin 0 or the LED 0
high. Right?>>So uh what it’s doing right now here is that we
uh took the tool, we disabled
the pin, and I’m uploading it right now, the rebug binary
which our tool did, uh the
GitHub link. And uh it’s gonna go ahead and we’re gonna go can
you reboot the router?>>Yeah.
Alright.>>So let’s look at what was the integrity status? Again,
this is on PRE-ROMmon, we have
full access now, FPGA cannot do anything and here it is. We have
updated the FPGA, it still
booted. You cannot apply any more patch. I have 20 SPI
flashes right now on my so I
have to like literally take it out, update it.>>Ta-da! Look
this is where you clap!>>Come
on guys->>Like, it’s very good!>>This is really cool though.
>>Right so [applause] now we’ve
left the output right of this thing saying that something is
wrong, right? In ROMmon so we
can demonstrate proof that okay so the FPGA is doing its job, it
just physically can’t reset that
processor and now we can actually patch ROMmon and
PRE-ROMmon to make this little
message go away. Right so but for the purposes of the demo you
know we’re showing that this
thing is trying to cry out for help, but no one is coming. We
win.>>And we’re gonna wait 100
seconds also while I show the other demo. Uh so as you can see
uh there were 4 lights before
and we have disabled the 1 light which was on the- uh at the end.
So again, try it out, play with
it, and uh what we want to->>Yeah and the reason why we
used the Mojo is that you know
this is a very inexpensive uh piece of hardware that has you
know the- the Spartan FPGA
inside so uh if you look- take a look at our tool, right, you
should be able to you know
modify this yourself and play with all of the different
options of taking you know pin
high, pin low, input to output and all of that good stuff. So
you know I think we’re uh about
out of time.>>Yeah this is the impact.>>Well right. Cisco
routers are used a lot in a lot
of places. Um but so we do make a lot of recommendations uh and
a lot more detail on the
technical side of all of this and the implications of other
things that’s not just a Cisco
router, right. In our- in our paper which is coming out in- in
a day I think or 2 days, uh at
usenix woot so you know we don’t have time to talk about all of
the recommendations that we have
for uh you know improving FPGA security right? Making it as- as
good as it can because
fundamentally this is something that is stuck in hardware. And I
don’t believe that this is
something that can actually be fixed via a software patch. So
please check out our website, uh
our code on GitHub, read our paper, read the f*cking paper
right! And um thank you.>>There
are 20 more slides. You guys can look at it later on.>>Yeah? I
don’t know I think we’re- we’re
getting kicked out. Oh! In order to reward you guys for- for
coming to the right talk- I’m
gonna throw some T-shirts! I mean who- who else is doing that
right? [crowd cheers] There you
go. You guys made the right decision in life and you get
awarded for it.>>Yay!>>So
[chuckles] alright well anyway thank you very much uh that’s
our talk. [applause]