During my univesrity studies, I applied my skills in several domains. Below are some selected projects.
Master Thesis (2024-2025)
- Title: Out-of-distribution detection in medical imagery using Zero-Shot CLIP based Vision Language Models
- Supervisors: Dr. Alexia Briassouli, Dr. Faizan Ahmed.
- The link to the paper can be found here.
I finished my master thesis at the University of Twente. It revolved around the detection of unexpected images (Out-of-Distribution) in medical imagery using CLIP-based Vision Language Models.
CLIP models are, at the time of writing, among the state-of-the-art models when it comes to multi-modal models, and have shown great aptitude when applied to many text-image pairs across many different domains. This generalisibility makes them adept at recognising data that is outside of its original (training) distribution as well. However, medical data is often more complicated to assess and baseline CLIP is not at a sufficient level to be applied to this task, which resulted in researchers developing medical variants of CLIP, like BiomedCLIP.
Noting the difficulties for AI present in medical imagery, I set out to assess whether these two models were apt at OOD detection tasks under different circumstances, such as different distribution shifts, distances to the ID dataset or medical domains!
The nuances regarding Out-of-Distribution (OOD) are very prevalent, but I managed to set up a pipeline to test these two models across several different medical datasets, from several different public datasets.
In the end, I found that the distance of OOD samples to samples in the In-Distribution (ID) was the best indicator for the performance of both models, and while BiomedCLIP did outperform CLIP on average, this difference was very small.
In the future, it is recommended researchers to consider models with more resources and to fine-tune models on data, in such a way that the OOD understanding of the models improves, especially in near-OOD scenarios. Moreover, this study was broad, so in addition, it is suggested to limit the scope of the experiments done as well.
Internship: Tree Species detection using YOLOv8 and UAV imagery (2024)
Within the Smart Lab of the Yamagata University in Tsuruoka, Japan, I set out to improve upon a solution for a problem the lab had been trying to tackle: the automatic recognition of tree species from images acquired by Unmanned Aerial Vehicles (UAVs). Due to Japan’s mountainous terrain, it is often difficult to do this classification by hand, and it makes much more sense to use drones to do this task semi-automatically.
During this project, I worked with several different technologies. From designing flight paths for the drones, to operating them during expeditions and from rendering orthomosaics of forests, to training a YOLOv8 model with the data, attempting state-of-the-art encoding of NDVI and other indices into image alpha channels, though that quickly became out of scope.
I also assisted other members of the lab with their projects using the skills I had cultivated previously.
As for the results of this project, I found that several features in the data could become difficult for the model to deal with, as even with more data, the model was not performing as expected on new forest test sites with the same species.
I surmised that this was caused by the differences in the features of the images depicting the trees still differs too much between sites to really be considered generalisable. In the end, it would have been better if data from more test sites would have been collected for better generalisability to other forests, but the performance of the model was not terrible on the site it was trained on.
It was able to localise the trees quite well, achieving mean Average Precision scores close to 80% when considering the standard Intersection over Union threshold of 0.5.
Bachelor Design Project (2022)
- DumpingMapper – Detection and Classification of Illegal Dumping Using Deep Learning and Computer Vision
One part of the final graduation requirements for the Bachelor of Computer Science at the UT is to do a Design Project, in which students form a team to realise a piece of software for a specific purpose, possibly initiated by a third-party company.
In my case, I worked together wit a group of 4 other students to realise a system to detect illegally dumped trash in Cyprus, specifically through the use of satellite imagery and computer vision.
Due to the size of this project, each of us worked on a different part of the system: the data annotation/synthetic data creation, a Convolutional Neural Network dumping detection, a Convolutional Neural Network dumping classification model, a database to store these detected dumping sites in and a front-end dashboard to display information regarding detected dumping sites.
I was responsible for the data annotation and synthetic data creation, arguably the most important part of the system. After all, garbage in is garbage out!
During this project, I devised a standardised set of rules for the annotation of the data (which was way too much to do on my own) and annotated around 4300 images in the span of 10 weeks. Additionally, I created roughly 100 different images synthetically using Blender code.
Using free Blender models and textures from the internet, I created several different materials, from soda cans to wooden planks to cardboard boxes, and placed them on a plane with the satellite image crops displaying no dumping. The approach aimed to simulate dumping, in such a way that the material of the dumping (wood, metal, cardboard, etc.) would be known before hand, because the spatial resolution of spatial imagery is often too coarse to reliably classify by the human eye in actual dumping sites.
Care was taken to make sure that the objects were placed on a semi realistic scale as well.
Surprisingly, the system performed quite well, achieving accuracy scores of 96% for the detection of dumping, partially thanks to the data I annotated.
The classification model was noticably less performant, but that is mainly because the amount of data that was put into this model was subpar, as a result of difficulties with Blender and the scarcity of different types of objects. But all in all, it was a good project, in which I learnt a lot and it holds a special place in my heart, since this is when I really fell in love with Data Science as a field.
Bachelor Thesis (2022)
- Title: Real-time Pitch Detection using a resource constrained IoT Device
- The link to the paper can be found here.
Some musicians struggle with identifying frequencies and musical notes in a songs. In this research, I created a solution powered by AI: a set of models designed to tell a musician the notes or chords they are playing, through an Internet of Things (IoT) Device, in this case a phone. If one were to describe it very crudely, it would be an AI powered tuner, with the possibility of identifying chords based on the frequencies present in the musician’s playing.
IoT Devices are often lacking in resources as opposed to other machines. Compare the computational power of a phone with that of a desktop computer for example, and therefore it is important to also work with models that can be used via the cloud. To achieve this application, I used Edge Impulse; a service hosting several models that one can train for different purposes, without the specific need to dive into the code for specific adjustments.
I made sure to include several different instrument timbres and frequencies, all based in musical understanding, to combine the models in such a way that it could predict the notes, intervals or chords present in any piece of music, using Fourier Transforms and Mel Frequency Cepstrums/Spectrograms to transform the data into model decodable form. The models were combined cleverly to allow the identification of the root note and type of major or minor chord in order to create a holistic and informed decision on the chord being played.
The tool’s performance with respect to chords or multiple notes at the same time can be improved due to the fact that Fourier analysis struggles with multiple frequencies in general.
