Yeah. Okay, everyone. So welcome to another spatial biology master class. I’m here today with Rachel Penny, who is a scientific officer at the Cancer Research UK Scotland Institute.

And we are gonna start with some questions I mean, polls.

We are testing this, so I hope it works.

And that also gives some time for people to start joining.

And then Rachel is gonna start her presentation very soon.

So we are just curious to hear a little bit about the background of our audience today, and we’d like to know what is your biggest challenge with cellular segmentation and multiplex image analysis?

I’m gonna give a few seconds for everyone to, yeah, vote.

And, in the meantime, just so you know, Rachel is gonna be presenting today a novel phenotyping method to overcome this challenge with cellular segmentation. So stay tuned.

Okay. I’m just gonna give, like, ten seconds, and then we can start.

Okay. So, Rachel, I’m gonna hand it over to you, and let’s get started.

Yeah. Hi, everyone. And let me just share my screen. Can everyone see my screen?

Yes. We can see your PowerPoint. Yeah. Now we see the full slide. Yeah.

Yep. We can see the presenter mode and anything. It’s just the slightest there.

Okay. Yes. Correct.

Okay. Well, hi. I am Rachel. Thanks for having me today.

I work in Cancer Research UK Scotland Institute as a translational histologist and the deep phenotype and advanced technology facility and professor John Lacan’s lab. And today, I’m going to be speaking about our method that we’ve developed, which is a global phenotyping method using the artificial the artificial intelligence module and Visiopharm to overcome the cell segmentation bottleneck.

But first, just a brief overview on some of the spatial technologies that we have. Some we we use Visiopharm to analyze and some we need to try or we haven’t tested out yet. So we have the ten x genomics xenium, which, we can do up to five thousand, RNA.

We have the Lunaford com oh, we don’t have the Uniphore comet yet the Lunaford comet yet. Sorry.

We are anticipating its arrival, but we’re hoping to get that quite soon. This can do more or less up to one hundred and twelve teams, maybe more, depends on how much we want to push it to its limits.

We have the acoia phenocycler fusion, and currently we’ve done around forty three proteins.

We have the Roche Montana Discovery Ultra, which is our lower clicks instrument, but it’s a more high throughput.

And this can draw up to eight proteins plus DAPI, and we scan all of our lower clicks, assays on the Acolya PhenoImager.

And our newest toy in the lab is the Olympus BS two hundred.

And today, I’m mostly going to be speaking about how we use Visiopharm to analyze more of our lower plex, assays.

So most of the tissues that we analyze, are from our lattice cohort, specifically our lattice a cohort, which is, around ten twenty five long adenocarcinoma patients across twenty three TMA blocks, and each TMA has three cores per patient. So, you can see here that the patients are in tranclic at with the controls so we can orientate it. And for each patient, we have sequencing data. We have all the h and e’s, and we have resection samples as well. So we have a lot of information that we can, use along with the Visiopharm data.

So even with our lower plex assays or the ones that we, or the images that we generate from, the Ventana and the Aquio female imager, and we then spectrally unmix them using the AquioInform with project specific spectral libraries.

And even though they are our lower plex, we generate highly data rich images, and I’ll talk you through how we analyze most of our images tissue segmentation algorithms.

We’ve developed loads based on the tissues and the projects that we’re working on. So we’ve got tissue segmentation algorithms for pancreas, kidney, lung, and liver, you name it. We’ve most likely got a tissue segmentation algorithm, trained to to segment the tissue. And on the right hand side here, this is a core with and that’s been stained with our immune panel, and it’s all long antimal carcinoma.

And we use, Visiopharm again, the unit to segment all of our images into background, which you can see outlined in green tumor and that you can see in blue here and red, which is stroma. And we also segment necrosis as well, but it’s just not shown in this core.

And we usually train, with human annotations using DAPI, autofluorescence, keratin as the input.

We will usually adjust the freeze depth as we see fit, and we try and aim for a loss of below zero point one. And we’ll always get the we’ll get two pathologists on our team, John Lacaine, who’s the academic lead. He’s, a trained, pathologist that he will, like I said, look at our images for us, make sure that our ground truth has in fact the ground truth. And we also have Kai who is a pathologist who is doing his clinical fellowship with us just now as well.

Once we segmented our tissue into regions, we then go into nuclear segmentation, and we use, the app that developed.

And we have retrained on a lot of our lung adenocarcinoma cohort. This is due to, our low DAPI staining in this tissue.

And, again, we’ll get our pathologist just to make sure and validate that everything’s okay.

So you can see here that it actually works quite well. It’s worked better than the mucinum, cue path, etcetera. So, yeah, we’re quite happy with it.

And for the nuclear for this wholesale segmentation, we like other segment ourselves based on the expansion of the nucleus, and we do this by ROI. So if a nucleus is setting within a tumor region of interest, we’ll expand by, say, ten to twenty pixels. And if it’s setting stroma, we’ll do maybe five to ten just depending on the tissue.

And we are interested in cellular phenotyping for, you know, biomarker discovery. So we have used the phenotype typing module and Visiopharm to do this. It’s based on the nuclear segmentation, and then we do a wholesale expansion.

So what we’ve just seen previously here as we’ll just turn the cytoplasm label just to one wholesale label.

And then the phenotyping module and Visopharm, uses the probabilistic model for clustering.

So for example, if you have CD3 positive cells, they will be a more numerous class as as CD3 and CD8, for instance, and they would form a subclass.

We find this a little bit challenging mostly because, when you have a cell that’s expanded, onto a different stain, for instance, it can be misclassified.

And when you’re training, the phenotyping module, you need to weight it, equally or you can mask phenotypes again or you can introduce more negative and an unidentified. So, yeah, we will use this as well, and we’ll do more, kinda standard methods by taking the main intensities of each marker.

And we do this by, again, using the phenotype guide the Phenooplex guided workflow. We think that this has actually been an asset that’s been introduced to Visiopharm for us.

So, again, our workflow stays the same. We segment the tissue. We segment the nuclei, and then we expand on the cell.

And then we manually set the, the threshold cut off ourselves, which is a feature that we really like.

So we can base it on the whole cohort or a pair TMA, for instance. And then it generates a phenotype, list, which is quite good.

However, because we have done the wholesale expansion and it’s not as accurate, it’s just based on, expansion of the nucleus, it can introduce false phenotypes.

So for instance, you can see here that that cell has actually been expanded onto well, it’s a size keratin positive cell a lot of p sixty seven in there, but it’s actually been expanded onto an SMA.

So this has case to false phenotype.

So we do feel that these methods are really good and are used all over the world.

But sometimes they can we feel they can be suboptimal, with phenotyping, and it’s limited to the accurate or the inaccuracy of the cellular segmentation.

Other labs, we know we use cocktails of antibodies to generate membrane markers, and that can be used for the cell boundary. But for us, especially for our more lower plex assays, this takes up precious space and our and our assays and our multiplexes. So we basically want a better way that we can phenotype without really accurately or as accurately as possible segmenting the cells.

So we have developed a method in collaboration with David Mason from Visiopharm, and it’s called Pixelmap. And I might refer to it sometimes as a global method as well. We’re just playing around with names now. But yeah. So this is Pixelmap or our global method. And this is our approach to tackle the ongoing challenge with accurate cellular segmentation and, therefore, cellular phenotyping. Because if you’ve got more accurately segmented cells, you’ll have more accurately phenotyped cells, and then you’ll have more accurate spatial analysis for biomarker discovery.

So a brief overview.

This is supervised AI using, the unit in Visiopharm.

It does not rely on accurate cellular segmentation.

The training images contain human annotations for entire cellular regions, and it’s learned patterns of positivity for each cellular compartment, and that is probably key in this method. So I’ll go into more detail about why we did this.

So this is just our, training pipeline overview.

So to get our training set, we got our associate scientists and the team to, extract single channel images from multiplex images. We use four different multiplex images in total on our long adenocarcinoma cohort.

And then just as a little bit of housekeeping within, vesiofirma, then sec like, separated them into different subfold to of cytoplasmic staining, membranous staining, and nuclear staining.

And and so then we annotated single channel images. So these single channel images that’s been extracted from the multiplex images, for the each cellular compartment, either creating a positive label where I e where the stain was localized to, or where the stain was or what was positive for that specific marker. So that would be positive. And then background. So our outputs would for the training would just be positive in background.

And then the pathologists and the team, so John invalidated, the label training set and Kai did as well. So the two of them sat down, looked at our images, made sure they were happy before we started started training.

And then this part is also key. So we trained all the images and the slide tree at the time. I know we don’t have a slide tree now, but, you would just highlight them all, and they were trained simultaneously.

And this is because they were extracted from the multiplex images, so they only have one channel.

And then we checked the app on, multiplex images that it hadn’t seen before just to make sure we were happy. So if we wanted to have a look at how well it could call a CD four positive cell, for instance, in a multiplex, we would go into the input channel and see CD four, and then we would look at the feature heat map. And then that’s how we validated it. The pathologist did the same. Pathologist had a look and made sure they were happy with it. And then the outcome was our affiliate app that contained information on regional positivity for each cellular compartment.

So this is just an example of our training images. So for cytoplasmic staining, we extracted pan sized keratin from this multiplex image at the top here.

We extracted phosphate eIF two alpha from the second multiplex image, and we extracted RPS six from the third. So this generated our cytoplasmic, cellular localization for the training set.

And for membranous, we’ve got the same multiplex image here, with definite core, though.

We extracted CD four channel from that. We extracted the CD eight channel from that, and we extracted CD nine SCA nine, sorry, and PDL one.

And for our nuclear staining, we use key sixty seven and FOX p three.

And I’m just gonna run into a little bit more detail, more zoomed down so we can see.

So where Magenta is, everything was labeled as positive, not just where the same is localized to. Like, if we had, for instance, I always use this analogy. If we had a doughnut, the whole doughnut was given even where the hole is and not just the ring of the outside.

And then we did the same thing from embarose veins, this is CD four and for k sixty seven. And here you can just see how, just a little bit housekeeping and Visiopharm, how I separated them into subfolders.

So the reason for doing this is because so this is a really bad drawing, but you can see that if we’ve get CD8 staining around the outside here in yellow, and then you’ve got the nucleus that would be in the middle. And see, we just add a nuclear segmentation, for example.

And mask or label would be setting inside this CDH stain. And because it’s not actually touching where the stain is, if we were to just take the mean intensity of that, it would be zero.

Even if we use a hundred percent of the object, it’s not touching it so it would be zero.

So the reason that we gave, the entire, region of positivity is being positive And so that even if we run a nuclear segmentation because it’s setting in the middle and that’s been classed as positive, then our algorithm is gonna say, okay. This cell or this label belongs to a positive, or as positive for CD eight, for example.

So that was the rationale behind the training labels.

And then all the images were trained simultaneously at the same so at the same time, and it doesn’t really matter, and within the AI architecture what you put as your input as long as it was index one. And this is because, like I said, all of the images that we’re training on only has one channel anyway. So it’s gonna use all of that information. It’s gonna look at what a cytoplasmic stain looks like, what a nuclear stain looks like, and what a membrane stain looks like. It doesn’t really matter what the stain or what the marker is. It’s just looking at the pattern, essentially.

And then we trained it over five hundred thousand times with a good loss. We adjusted our freeze depth every twenty or thirty thousand iterations as we seem fit.

And yeah.

So this is the output. So we then generate a feature probability heat map, which can be then extracted from multiplex images. So your feature outputs would be, backgrounds or other and positive.

So on the right hand side here, you can see our nice immune panel image.

And within the AI architecture, within our pixel map or global methods, we have inputted our key six to seven because we wanted to see if it could correctly extract the probability heat map for key six to seven.

And on the left here, this is just the multiplex image with just k six seven turned on just so we can compare and see if it was correct. And you can see here that the k six seven probability heat map is actually really quite accurate. It’s really nice.

And we did the same thing for our CD six to eight.

Same thing for CD eight.

So you can see that it’s classing not just where the stain is localized to, but the whole cell, CD four, and pan cytokeratin.

So the heat map shows the likelihood of each pixel that is part of a cell that’s positive for the marker that you select within the input.

And then we wanted to try we basically wanted to try and push it to its limits and see what we could get from pixel map or the global method.

And we this is an image of an organelles panel that we, did as a lab that we optimized.

And and the kind of pink here, the light pink, you can see the cocktail of c d ninety eight and eighty q one a.

And by just inputting that channel within the AI architecture and review the positive feature, you can actually see that it’s got the, area of positivity really, really accurate despite not being given a cocktail marker within, the training. It’s just learned that this looks the same. This region of positivity looks the same as what was given in the training set. So it thinks it is positive for that corresponding marker.

And you can see here that this is actually despite Pixelmab only being trained on lung adenocarcinoma of human species, this is a lung a mouse liver tissue, liver cancer tissue, actually.

And you can see down the left hand side here, this is just the original channels from this section here of multiplex with just the channel and DAPI turned on. And then on the right hand side, this is a probability heat map for the corresponding marker. So you can see that it’s actually extracted the probability heat maps for each marker really, really well despite not being given, MAC and CD forty five and the training set and despite not being given actually any mouse tissue. And the reason for this is because it’s learned the pattern of positivity for the cellular compartment, so it knows.

And we found this really, interesting as well. So in the middle here, this is, from another six plex, but we’ve just got DAPI CD four and CD eight turned on just for, like, good visualization.

And by inputting the CD four channel within the architecture and review the probability heat map for CD four, it can pick it out really, really well. And when you input the c d eights into the AI architecture, it extracts the c d eight probability heat map really well as well. Despite both, cells looking the exact same and there being similar morphologies, it knows the difference between the two because you’ve inputted the channel into the AI architecture, and it’s just used the regional pattern information to extract the heat map.

And what we think is really impressive here as well is this is, another multiplex panel that we’ve done just showing CD sixty eight in pancycan dappy turned on. So we’ve got, pancygeketin in green and CD sixty eight in red. And we just inputted CD sixty eight within the AI architecture and then reviewed the positive probability heat map for CD sixty eight. And, again, it’s out really quite well despite actually not using an AcuFeed marker within the training. It albeit as a cytoplasmic marker, but it’s quite messy. But it’s still, being able to accurately extract the heat map from this image.

So nipexomap contains information of regions of positivity, for each cellular compartment in free cellular marker.

And it can extract phenotype phenotypic heat maps or probability heat maps from multiplex images.

We want to know or we wanted to first know how we can actually use the probability heat maps for the phenotyping and have we really mitigated the need for wholesale expansion? This was the main focus point here. Have we really mitigated the need for wholesale expansion because the entire regions will be classed as positive for sales?

So we did a minimum object size test, and we did this just using, the pan sized keratin phenotype assignment.

And we had four test groups of different nuclear sizes, and we compared this to manual assignment of size keratin positive cells. And it was the pathologist Kai and the team that did this for us.

So to begin, we did, an object detection.

And that’s to start with was just a nucleus only detection.

And that’s what’s just to do that. You can see in blue here that we’ve got the nuclear masks that have just been detected to detect the nuclei.

And then because we’re actually gonna be dilating and or eroding each nuclear mask, we want you to make sure that the x and y coordinates remain the same throughout the test groups.

And we did this by using, the outline by an with annotation tool and the post processing steps.

So we did that, and then we copied each of the images into the four different folders corresponding to the nuclear size test groups. So we had nucleus plus three pixels, so we dilated the nucleus by three pixels. We dilated nucleus by eight pixels to try and generate a wholesale label and the one pixel test group.

So you can see here that, again, we dilated by three pixels, eight pixels, and I don’t know if you can see it quite clearly, but in the middle of each pink annotation, we’ve got one pixel spot and the blue spot as a label.

And then what we did was we used the, global phenotyping app or pixel map with the deep learning. We went into the eye architecture, turned it, put the channels and put channel or index one to size keratin. We then turned the, app to then sorry, an old classification.

And then we went into the post processing steps and said, change my intensity label cell or this blue label that we have here.

Change replace with a new positive label, which you can see here as in yellow. If it’s fifty percent, so zero point five or fifty percent sure that it belongs to the positive feature or the positive, feature for, cytokeratin, which we thought worked actually quite well just even by looking at it, by eye.

And then we generated the x y coordinates of all of the nuclear annotations So they would all stay the same so we could then compare them.

We took the object and for each of the labels, so we knew if it was a positive cell or it was just an active cell, and we took the counts as well.

So just for QC purposes, mainly for this point, we use the QC tool in Visiopharm.

And I just filtered by the x and y coordinates, create this tesny, and then filtered by the positive label. So what was positive for cytoskeleton and what for by the ground truth and what was or what pyxomap seemed to be a positive label.

And just looking by eye, you can see the ground truth had a hundred and twenty one positive cells within that specific ROI. And then you can see here by the looks of it that the one classification was actually next best in line. So that was the best test group apparently. And then you can see here that by eye, you can see that the nuclear detection looked to be the second best. So all the x y coordinates lined up perfectly. All was well. So we got Kai, the pathologist, to to do a confusing matrix just to make sure that what we’re seeing is fine, and we might not need to expand, on the nucleus to create a cell.

And you can see here that the confusion matrices from a here and to d here And comparing nuclear detection, the dilate by three pixels, dilate by eight pixels, and the one pixel group comparing to the ground truth. You can actually see here that the nuclear detection only, so just a label on the nucleus was the best and and was more compatible to our ground truth annotations.

You can see the next, best was three pixels, and then dilate for eight pixels wasn’t very good at all. And what I don’t actually expect, the one pixel was the worst.

So by doing that, have we, oh, to begin with, how do we use the probability heat mask for phenotyping? You can do it that way like I did. So you could do the post processing steps depending on what you put as your input within the AI when you turn it the app tunnel classification.

You can say change my intensity of the label cell to, as you know, p six seven positive or CD eight positive. And depending on how many markers you want to phenotype, you would have that amount of that.

And then the next question that we had was obviously, have we really mitigated the need for wholesale expansion?

And to be honest, at this point, we would say yes.

We still to do the test with all the other, markers within the multiplex. But by eye and looking by with, Kai and John just looking at images, they would agree, yes. We have mitigated the need for wholesale expansion. We can just use the nuclear label.

And then next would be the question was, what what’s the output? Like, what can we get from the heat maps?

And so we go back to the beginning. So we segment our tissue and to tumor, stroma, background, and necrosis.

We then just do the nuclear segmentation.

Just plain and easy, plain and simple, nuclear segmentation, and then the output.

So we would have our global method or our pixel map, and then this would be used as a parent app. And then depending on what you have as your input channel, so for instance, we we have CD four at the moment. We would then turn the CD four app to an old classification app.

And then as out and the output variables, we would then see the main intensity of the positive feature, so the probability of CD four, in each label, so in each nucleus or each cell, whatever you want to call it. And then depending on how many, phenotypes you want to get out of this, you would have that corresponding to number of apps.

And then the output would then be your probabilities.

So you just get your standard, TSP that you can change or you can open in Excel. And then for each phenotype, you will then be given a probability based on if the nucleus if Pyxelmab deemed this to be positive or not. And then you could set your probability cut off so you could set how confident you want to be if you want to be fifty percent and above seventy percent.

We are still currently benchmarking this. So we were benchmarking against normal mean intensity measures that we would normally do, and comparing this to the ground truth annotations of what kind, John are gonna be phenotyping.

But we are currently benchmarking what is the best how confident we need to be for this to be accurate, essentially.

So for future work, we obviously are pushing to publish.

We’re hoping to have this paper out in plus by the end of the year.

And we want to publish data from, our other multiplex panels and our more high plex panels because Pixma was trained or the global method was trained just on the, images that was generated from the pheno imager. So and we’ve tested this on the comet as well, the one for comet images, and it works quite well.

Probably works on about eighty percent in the marker, so I think there’s some room for improvement in that instance anyway.

So we want to try and get Pixelmab to work with more high plex acids. So for instance, the femocyclic or fusion.

We want to refine Pixelmab for fibroblast phenotyping. So because, we don’t actually give Pixelmab the, morphology of the fibroblasts with the long elongated cells, it’s not very good at identifying fibroblasts as of yet. That’s not to say that it won’t be able to do it. I strongly believe that we can.

We also want to try and incorporate some RNA scope or spot analysis. We know that this can be done. We just need to do it within our method.

And yeah. So cross imaging platform method, we want to try and get it to work on, for instance, theon cyclo fusion, etcetera.

So thank you so much for listening. I’d like to give a massive thanks to the Lacan lab, and specifically Leah, Ian, and Kai, and David Mason from Visiopharm and the Tom Bird Lab for allowing us to use, their energies well, our images that we developed for their, project.

So thank you. I’d love to answer any questions if you have any, if we actually have time.

Thanks, Rachel, for this great presentation and sharing. Well, congratulations for developing Pixelmap.

And, well, we look forward to see the method published soon so it can inspire a lot of other researchers around to start using it.

So let’s start with the q and a.

So we have a first question from Shannon.

Has this workflow been tested on imaging mass cytometry data or only fluorescence images?

Currently, we’ve only done it on fluorescence just now for protein based. And, again, we’ve mostly tested it on the pheno imager. So the imaging methods that you extract autofluorescence and uses it as a channel.

So, yeah, we’ve only done it on that, but that’s a really interesting point that we could hopefully test in the future or, by all means, you test it, but not before we publish, please.

Cool.

And the next question from Shatavisha.

For the probability heat map, how are the thresholds for positivity calling deterrent?

How can this be adjusted?

So, you know, when you, get an AI or trained, deep learning algorithm in Visiopharm, you can look at the features and you hover over to see if it’s, if you hover over with it’s white, that’s, like, ninety to a hundred percent positive. And if it’s black, it’s zero.

So right now, we just use a fifty percent probability for the heat maps just as standard just now. And then, like I said, we are currently benchmarking pixel map against standard methods, like, just using immunotensities to phenotype and bimanual assignment of the phenotypes.

And we’re going to try and see how harsh we can be to still generate accurate phenotypes. So putting the confidence or the probability up to seventy percent, eighty percent and see, yeah, how harsh we can be, but still not lose accuracy, essentially. So right now, our pathologist is doing that.

But yeah.

Thank you. And Sylvia is saying, hi. Great talk. Why do you think at the end, first picture was the worst with the confusion mattresses when it looked better before?

Okay. So when we generated the test name, so because I was just looking I was just looking at the x and y, so we’re just looking at where they were, like, where all the sales were.

It’s only comparing the number of positive sales. It’s not comparing if it’s actually right or not. So it could be a a hundred and twelve I can’t remember what the number was. It could be a hundred and twelve incorrect to classified side skeleton positive cells.

So, obviously, at first glance, I was like, wow. This looks great. Like, one pixel can be the best, but turns out with the confidence matrix that, CHI generated, no. The nuclear detection was the best compared to the ground truth.

Thank you. Alessandra is asking, so phenotyping ultimately results from a decision on an arbitrary probability value? Mhmm.

Sorry. Yeah.

So phenotyping ultimately results from a decision on an arbitrary probability value? I think that’s So, yes, essentially.

So we would look at all the probabilities generated and then set the confidence or set the cutoff and say, yes. This This is correct.

But you I suppose you would do that with generating mean intensities for wholesales anyway. You would look at your data. Yes. Do you there would be some normalization methods there, but you’re still looking at the mean intensities. And sometimes the mean intensities can actually quantify depending on the scanners and stuff. So, I mean, I would I’m not saying by any means that it’s gonna overtake the intensity, generation of phenotypes.

For we’re just saying for us right now, it seems to be working pretty well and as compatible with, ground truth annotations. So, yeah, we do think it’s working quite well.

Great. Question from Sylvia. Why do you not annotate the whole cell when training for nuclear positivity?

We don’t annotate the full cell when training for nuclear positivity because we’re using it with a nuclear segmentation app anyway to generate, nuclear labels. So it there wouldn’t really be much point because the cell would then become positive anyway of it sitting on a positive nuclear label.

Great. And question from Matthew. Thanks for the talk. How do you go about extracting single channel images from your, MIF images?

Very, very good question. I only know that Ian used to do that.

But if you’re interested, you can put me an email and I can copy Ian in if you want.

That’s beyond my realms of understanding at this point because that’s scripting.

But, yeah, Ian did in Qeopath.

Cool.

Thanks, Rachel, and thanks, everybody, for joining today. This was our last edition of the year. We are hoping to continue next year with more speakers.

But thanks, everyone, for joining. Thanks thanks thanks to you, Rachel, for sharing the method with us, and your presentation today was great. And, yeah, I we are not gonna share the recording yet because, the method has to be published first. But as soon as it’s live, we’re gonna, yeah, share it and so you can watch again.

Yeah. Thanks, everybody. Thank you.

And have a good end of day.

Have a good Christmas, everyone.

Oh, yeah.

True. Happy Christmas. Bye.