Introduction

A potential use for automated object identification would be to extract a measure of the number of birds or squirrels visiting a bird table or feeded over the course of a day. There are likely to be large portions of the video with no activity at all. An automated technique can both collect data directly and also screen for interesting portions of the video if the study includes observations on behavious.

This you tube link is provided as entertainment for cats and shows constant bird table activity.

https://www.youtube.com/watch?v=yWE_QPdtatY

This is the result of object identification.

http://r.bournemouth.ac.uk:82/examples/yolo/pred/animated2_bird_table.gif

Extracting frames

The first step involved in classifying a video taken from a web cam is to extract individual frames. There are various techniques to do this using a range of software. Some use graphical user interfaces. The video viewer VLC can be set up to extract frames as the video is viewed. However this takes as long to run as the video takes to watch, so does not speed up the process of screening for activity. On Linux ffmpeg can be called from R to extract stills. The frame rate chosen is 1/number of seconds between still. So the following code extracts a still every five seconds from the start of the bird table video.

### Set up parameters. 
path<-"/home/rstudio/webpages/examples/yolo/video"
nm<-"bird_table"
frame_rate <- 1/5
video_file <-sprintf("%s.webm",nm)
out<-paste(nm,"%04d.png",sep="_")

If the video file consists of a long time period the frame rate may be set to a smaller fraction. Capturing a frame every thirty seconds may be sufficient to detect feeding on a bird table. As the time taken to classify each image is just under three seconds this would allow ten hours of video footage to be converted to a data table in around one hour.

com<-sprintf("ffmpeg -i %s/%s -r %s %s/%s",path,video_file,frame_rate,path,out)
  
system(com)

Classifying and extracting data

The code explained here https://rpubs.com/dgolicher/yolo can be reused to classify the series of still images.

This code saves each image with the prediction boxes. Over time this may use a lot of disk space and is probably unecessary. The main result would be the data table.

library(Rcpp)
cppFunction('void redir(){FILE* F=freopen("capture.txt","w+",stdout);}')
library(image.darknet)
detect <- image_darknet_model(type = 'detect', 
 model = 'tiny-yolo-voc.cfg', 
 weights = system.file(package='image.darknet', 'models', 'tiny-yolo-voc.weights'), 
 labels = system.file(package='image.darknet', 'include', 'darknet', 'data', 'voc.names'))
a<-c(dir(path=path,pattern="png"),dir(path=path,pattern="jpg"))
a<-a[grepl(nm,a)]

f<-function(x){

  fl<-paste(path,x,sep="/")

  pred <- image_darknet_detect(file =fl, 
                          object = detect,
                          threshold = 0.19)
  system(sprintf("mv predictions.png pred/%s",x))
  pred
}

redir(); 
d<-lapply(a, f)
library(dplyr)
library(tidyr)
d<-data.frame(txt=unlist(readLines("capture.txt")))
d %>% filter(!grepl("Boxes",txt))->d
d %>% filter(!grepl("pandoc",txt))->d
d %>% filter(!grepl("unnamed",txt))->d
d$isfile<-grepl("/home",d$txt)
d$txt<-gsub(path,"",d$txt)
d$file<-ifelse(d$isfile,d$txt,NA)


d$object<-ifelse(!d$isfile,d$txt,NA)
tidyr::fill(d, "file") ->d
d<-na.omit(d)[,3:4]
d %>% separate(file, into=c("file","time"),sep=":")->d
d %>% separate(object, into=c("object","prob"),sep=":")->d
d %>% filter(!is.na(prob))->d
d$time<-aqm::clean(d$file)/frame_rate
library(ggplot2)
d %>% group_by(time) %>% filter(object=="bird") %>% count() %>%
  ggplot(aes(x=time,y=n)) +geom_point()+ ylim(0,6)+ylab("Counts of birds") + xlab("Time since start in seconds")

aqm::dt(d)

Animated gif of results

The animated gif can be produced from the stored images uses image magick. It is too large to embed within an rpubs document.

http://r.bournemouth.ac.uk:82/examples/yolo/pred/animated2_bird_table.gif

# path<-"/home/rstudio/webpages/examples/yolo/pred"
# com<-sprintf("convert -delay 200   -loop 0   %s/%s*.png   %s/animated_%s.gif",path,nm,path,nm)
# system(com)
# com<-sprintf("mogrify -layers optimize -fuzz 3%% -quality 10%%  %s/animated_%s.gif", path,nm)
# system(com)
# 
# knitr::include_graphics(sprintf("%s/animated_%s.gif",path,nm))