Charley Weber Portfolio

ANALOGUE VERSUS DIGITAL MEDIA

Ironically many of the best visual designers spend a lot of their time finding ways to make their digital assets look and feel more analogue, as that medium tends to offer more in the way of organic textures and lucky accidents which often form part of the visual designer’s toolset. However, that has more to do with personal aesthetics than any structural or data-driven design considerations. The benefits of digital communications are more structural, often primarily considered to be interactivity: communication between a user and author, as opposed to a passive experience of listening to an analogue Radio or TV. Other benefits include lossless reproduction, increased ability to manipulate, analyse and visualise data, the ability to scrape the net for digital time and location imprints, as well as 3D CAD animation and the ability to reconstruct scenes in non-fiction and narrative forms of journalism.

From working as a film and commercials editor, I am aware of how much work goes into producing even a minute of a good commercial, and am not inclined to suggest use of interactivity where it isn’t helpful or doesn’t serve some purpose of the story or design objectives.

Fig. 6 Developing Data-driven Reconstruction and Interactive Hybrid-Documentary Forms for Learning and Entertainment (Weber, 2022)

An illustration of this can be seen in my first short paper, see Fig. 6 Developing Data-driven Reconstruction and Interactive Hybrid-Documentary Forms for Learning and Entertainment (Weber, 2022). I wrote a paper and developed a practical model to examine the possibility of using limited forking narrative to observe and train for better outcomes and choices in dangerous or crisis situations, such as protests. A model I adapted from Augusto Boal’s (Maccioni, 2016) Forum Theatre methods, for behavioural observation and improvement of social, civic and commercial processes. The structure is mostly linear documentary in form, but at moments of crisis – in this example, towards the end of the timeline in Fig. 8 – the user is confronted with a set of choices as to how to proceed at a point of conflict and then presented with consequences of those choices. In order to complete the study, I would need to also explore the actions and effects of the opposing side in the conflict, in particularly the use of Kettling by the police and responses by law and order, which could be analysed to understand where there seemed too little security – during the storming of the Milbank – and over reach, which many would argue was the case in overly liberal use of Kettling the protestors.

Fig. 8. Flowchart for Developing Data-driven Reconstruction and Interactive Hybrid-Documentary Forms for Learning and Entertainment (Weber, 2022)

In this thesis practical model for the Ragamuffin App, I wanted to take that a step further by creating a self contained prototype app, which has similar structural and documentary elements, but is not based on the web or in the cloud, and instead downloadable from the app stores.

Fig. 9 Ragamuffin App site map (Weber, 2023: personal collection)

TRANSMEDIA FICTION & NON-FICTION

Next I will look at transmedia. According to Technopedia “A transmedia project may combine many different types of prints or prose text, graphics and animation, or work across multiple platforms, such as different types of social media platforms, interactive websites or advertising” (Rouse, 2019, p.1). The use of the word ‘may’ in the definition leaves ambiguity for interpretation, which particularly may be felt by non-fiction multimedia producers and writers. 

Transmedia  is nevertheless an area to consider if planning an enhanced app for non-fiction, and an approach which I used in my prototype app model to produce additive stories, told on different types of media from the original non-fiction form (book). As can be seen from Table 1, major transmedia titles appear prevalently in hugely popular major film, fantasy and science fiction titles. Jenkins’ theories on transmedia concerned themselves with works of fiction, gaming, films, comics, graphic novels and books. He describes the process of additive story as being built on characters and places on previous releases, adapting the existing story so that each iteration on a new platform imparts some new information. ‘’In The Matrix franchise, key bits of information are conveyed through three live action films, a series of animated shorts, two collections of comic book stories, and several video games.’’ (Jenkins, 2007, p 1-2). 

Table 1. Film and Fiction Titles in Transmedia  (Weber, 2023: Table from personal collection)

	Film	Book	Documentary	Podcast	Comics	Game
Star Wars	☑	☑	☑	☑	☑	☑
The Matrix	☑	☑	☑	☑	☑	☑

Jenkins (2007) defines transmedia as creating a storytelling world where key elements of a fiction get produced and distributed across a number of delivery channels ‘’for the purpose of creating a unified and coordinated entertainment experience. Ideally, each medium makes it own unique contribution to the unfolding of the story.’’ The question raises itself: were these the result of a transmedia design, or simply the result of hugely popular titles filtering into many areas of media, all eager to tell that popular story? At least with the advent of radio and TV, this seems to be a naturally occurring phenomenon.

Table 2.  Non-Fiction Titles and Transmedia  (Weber, 2023)

	Book	Film or TV show	End eBook or App	Web Doc		LinearNonlinear
The Great British Property Scandal		☑				Linear
Out My Window  Urban Culture. (2010)				☑		Nonlinear
World War 2			☑			Nonlinear
Conversations with JFK	☑		☑			LinearNonlinear
Blowout Mario (2012)
Ragamuffin’s Tale	☑		☑			NonlinearNonlinear
Mutante	☑			☑		LinearNonlinear
Bear 71				☑		Linear &Nonlinear

Looking at Table 2, Non-Fiction Titles and Transmedia, what presents itself is a far more patchy picture, which leads me to question whether non-fiction is naturally a more niche form and better suited to targeting selective audiences over one or two selected channels of distribution. 

However, there are powerful examples of the application of at least Convergence if not Transmedia principles, if for nothing more than using multiple channels, like TV and a central website to gather feedback which can then affect the outcome of the highlighted social issue. One TV project at the top of the table (Figure 5), The Great British Property Scandal came from the architect and TV presenter George Clarke of Amazing Spaces. He took a problematic area of social policy – lack of housing – and asked viewers to feed back into the TV program to report unoccupied housing. The level and immediacy of the response was remarkable. Over a hundred thousand people signed a petition and roughly eight thousand empty properties were reported by the public, many of which were put into use. “£17 million allocated for new national low-cost loan funds and  George Clarke, who had the original idea and hosted the show, was appointed Independent Empty Homes Advisor to the Government” (Alzamora and Tárcia, 2021, p. 59). Noteworthy is also that similar successes have been reported by the Red Nose Comedy charity project.

To me this shows the power of convergence across broadcast and the internet, non-fiction and modern journalism. The main takeaways in format for transmedia seem to be that, to follow Jenkins’ definition of transmedia, the form needs to include ways for readers or fans to interact and influence story, for example in the area of fanfiction or in the social sphere, in social journalism, for ways for the viewer to respond and act to influence the success of the non-fiction project.  

RESISTANCE TO TRANSMEDIA & INTERACTIVE DOCUMENTARIES 

In one major initiative by the Norwegian Film Institute to encourage the documentary producer’s community to engage with, explore and develop interactive documentaries, many of the producers appeared reluctant to engage the Norwegian Film Board’s attempts to encourage this area of production.  The reasons given were primarily because the producers were used to telling a story from beginning to end, in the traditional way. 

 But the important thing that is difficult for us, is to open up for user participation. We resist this, because instinctively we know that we have become what we have become because we tell stories to an audience, not because the audience tells stories to us. ”Interactive documentaries  are another world and not just one but multiple formats… I know how to deal in the film world, but I do not know how to deal with a programmer. I do not know what it costs”. (Karlsen, 2016. p 7)

So one factor holding back the development of transmedia documentaries may lie with the traditionally high costs and levels of expertise required to produce an interactive one, which may also require a level of coding skills to produce an interactive documentary with the polish and interaction required for a professional release. Because the producers don’t understand the skill set, let alone the production processes involved, the idea of producing an interactive documentary does seem to scare away producers of the traditional documentary. ‘’I do not think I know enough about that. If I am going to plan, fund and calculate the cost of doing a larger transmedia project, I will need to get some help, indeed’’ (Karlsen, 2016. p 7). Going some way to overcoming those objections may be creating clearer structural formats and mental models for both producers and users to understand, develop and engage with more easily. Another answer may lie in the form of low or no coding and Artificial Intelligence assisted coding for interactive and software creation.

WEB DOCUMENTARIES 

A form which has established itself more widely than mobile apps, in terms of non-fiction, is the Web Documentary, see the table  (Table 2) which now has multiple titles and its own unique category at a number of film festivals, including Tribeca and Sundance. These differentiate themselves from apps mostly in the delivery system, which is over the internet. One of the most successful and award winning webdocs, which can be seen in Figure 3, is Out My Window (Open Culture, 2010), which through testimonials, out of window views and pictures explores the reality of life in urban life and crime in and around highrise developments. For me, this is a socially relevant use of technology because it gets to the heart and truth of our experience, as opposed to purely escapist experiences, although that also has its place. Another one, Bear 71, (Allison, 2009) follows the life, trail and travails of a bear in the wild as human life encroaches on its territory. lt crosses over into the realm of environment, sensing and monitoring and so is interesting as a different approach to producing digital documentaries. Given the level of concern over the climate crises and the effects on wildlife, this type of open observation cameras and geospatial tracking crosses the realm of author/producer defined narrative and allows focus and the user to extract their own conclusions from a naturally verifiable story or set of events.

ENHANCED LEARNING TEXTBOOKS

Enhanced pedagogy textbooks currently exist, from publishers such as VitalSource, Cambridge University Press and Pearson. But these offer basic hypertext features, such as pop-up definitions, links to videos, references, extended articles and sources on the internet. Excellent for studying, in terms of coming up with a new model for broader non-fiction, they are far from the immersive type of multimedia experiences envisioned by, say, Arnaud Nourry or Dan Snow (Dredge, S. 2012) or, at an extreme, techno poets like William Gibson. Whether the interaction is sufficient to draw people into learning via a revolutionary new immersive experience seems unlikely. These are not  different to the early and existing forms of the enhanced ebook, which have so far not proved creative or imaginative enough, in terms of developing an attractive new form to draw people in without a captive learning audience, which students represent.

HISTORY & ANTIQUITY APPS

A great believer in multimedia  apps, over books, for the purposes of teaching history is Dan Snow, the historian and TV presenter. With several interactive history app releases including the Winchester App, a simple geopositioning app designed to guide history fans around the city’s key historical sites (Dredge, S. 2012). Another notable example of these includes WWII for the iPad (Dredge, 2012). Criticism which has surfaced regarding the World War II history app is that  “for all the claims of innovation and radical transformation, this is essentially old-style narrative history of great events in the Whig tradition” (Ralph and The Literary Platform, 2012). The writer goes on to say that of the two thousand historical ‘facts’ on which the app is based, appear to be the accepted perspectives one might find in an encyclopedia and which don’t offer anything by way of introducing new insight or understanding from the use of digital methods. If an engaging  animated history app were to contain narratives which also add to accepted and were able to challenge accepted state versions of history, in the way the Open Source agencies do, that might offer a truly unique, open and fascinating interactive new format for non-fiction history storytelling.

OPEN SOURCE REPORTING 

Although separate from the idea of enhancing nonfiction books with an app or digital augmentation, I have included open source agencies because the mechanisms for digital analysis and enhancement are fundamentally similar for enhancing a book and vary more by the degree to which the burden of proof is applied for different target audiences, for instance for the International Criminal Courts, in the case of Forensic Architecture.  

       Fig. 10. Gaza Hospital Bombing  Forensic Architecture’s Inital Analysis (Thomson, 2023)

Forensic Architecture is at the scientific, forensic end of digital analysis and storytelling and is one of the leading lights in digital reconstructions, particularly of human rights abuses and war crimes. Figure 10 shows  Forensic Architecture’s Initial Analysis of the Gaza Hospital Bombing (Thomson, 2023). Founded in 2010, Forensic Architecture breaks down their methodology into scientific forms of analysis of cracks and fractures, triangulation of telemetry, trajectories, matching landmarks and analysis of the angle of the sun and shadows to create an empirical argument, but usually about a personal story. ‘’They might scrape thousands of images of a bombing off social media and match them with material facts to fix facts in space and time, as if with the coordinates of a multidimensional map’’. (Moore, 2018, p. 2-3). 

Critics of the agency dismiss the group as “just artists” (Watlington, 2023, p. 3). This accusation has been leveled partly as a result of putting on exhibitions at museums and from being nominated for the Turner Prize. Weizman, who himself himself says “pro-Assad groups have sought to dismiss their work with Amnesty International exposing Syrian torture. [They] accused us of being ‘artists’, of being special effects, of fabricating the evidence”. My sense is that using data evidence when analysing and reconstructing a sequence of events can only help to illuminate the reality of actions of questionable actors in complicated human rights or environmental cases. 

Bellingcat, who it has to be noted came to a different conclusion on the hospital bombing, was founded in 2014 and is another open source investigative agency. The agency became widely known after their investigation into the Russian claim that Ukraine shot down flight MH17. Bellingcat proved the satellite images had been manipulated and Russia’s claim was therefore false.  One of their stated primary aims is to archive important data and stories while still available online, before they disappear or are erased from the digital world. 

The New York Times Visual Investigations Department was launched in April 2017 and represents an evolution of what was called “social journalism” around that time, as it applied to breaking news and trending stories, also referred to as the open source department. 

MECHANISMS TO ENHANCE NON-FICTION

I will look at more closely at the mechanisms to enhance non-fiction in this next part of the thesis, and include reconstructions of incident scenes using CAD, scraping the net for time and date images, qualitative and quantitative research, open source research using APIs, and the ability to build interactive database and types of web documents or apps – as opposed to simply publishing text based articles and photos. All of which can be employed by the digital producer – or their technically semi-literate editors – to reconstruct and analyse an event or sequence of events from current or recent history, a memoir, or from deeper history of an antiquities site.  

TEXT AND AUTHOR NOTES

To an avid fan or collector, annotations from various stages of a project’s development are an attractive enhanced edition feature, whether for a book, historical site or film. One enhanced edition of Ayn Rand’s Atlas Shrugged  contained various draught manuscript versions with personal notes made by the author (Britting, 2017). For fans of whatever ouvre, this offers added value and interest, particularly to collectors of limited or enhanced editions.

IMAGE ENHANCEMENT

 Figure 11 shows images from Picasso: The Making of Cubism 1912-1914, MOMA’s iPad app, which was released with the exhibition of the same name (Budiu, 2014). Although no longer available, the app offered various developmental iterations of the artist’s paintings taken from the exhibition, along with infrared analysis at different stages of the creative process. 

Fig. 11. Picasso: The Making of Cubism 1912-1914, MOMA (Budiu, 2014)

AR GRAPHICAL ANIMATIONS

Fig.12. Mutante (Source: Arrevabene, 2009)

One of the primary enhancements we can add to a book app, as Dr Rafael Arrivabene evidenced in his design book Mutante. In the limited release book and enhanced interactive AR Webdoc for this, his main undergrad design project, now Games Design Lecturer, Rafael Arrivabene, PhD created a book with finely typeset designs and symbols from historical cultures and esoteric philosophies (Dr. Arrrivabene, 2009). Inset from these were texts and designs taken from those traditions. For the enhanced web book he first asked: what types of media can digital offer which the book can’t. The first answer was AR triggered animated illustrations which replace the still illustrations on the page. While this might seem simple, very much like the Trinity and the Moma Apps, much in design depends on strength and simplicity of the concept and the execution, and in this case the choice of minimalistic but aesthetic choices in the context was impressive as a solution. 

AI BASED FACIAL REANIMATION

Ai is an area going through leaps and bounds in development, although controversial in terms of  copyright.  The thesis will confine itself to the use of cleared images and subtle animation of stills of family and characters in a memoir. The process illustrated in Figure 13 uses an AI technique called deep learning to reanimate faces. 

Fig. 13 & 14. AI family images (Weber, 2022: personal collection)  

One online user described the effect as “Harry Potter-ish but altogether mind blowing. Another one of my unknown WW1 soldiers brought back to life’’ (Hern, 2021, p 2,3).            

If you want to add talking audio to the animations, programmers, developers can access deepfake codes on GitHub (iperov, 2023, p 1). 

SOUND & AUDIO ENHANCEMENT

Writer, director or actor reads, documentary extracts and interviews add depth and richness to actuality based digital or web apps. Another enhanced mixed media book which has stood the test of time is Jacqueline Kennedy: Historic Conversations on Life with John F. Kennedy. ‘’The enhanced eBook includes a video clip at the beginning of each chapter that sets the scene’’ (Kennedy, 2011, p. 1).

The Ragamuffin App features an interactive multi-character cover picture matrix at the beginning of the application, as can be seen in Figure 15.  When a user clicks on a character, the app plays a sound byte which reveals something about that character’s personality, much like the program handed out at the start of a theatrical play. So for Dr. Timothy Leary can be heard his ‘Tune in and drop out’ speech, for Keith Richards, we hear him talking about the British blues invasion as a positive rather than a negative impact, on the issue of appropriation.  Additionally, for the app a narration character was created of an Australian, a necessarily loud narrator and  helicopter pilot. With sound effects and audio added.                      

Fig. 15. Ragamuffin App Characters (Weber, 2023c)

SATELLITE PHOTOGRAMMETRY

Available through online earth mapping services, like Google Earth and Bing, can offer the digital journalist and enhanced book app producer stunning simulations for coverage of locations, overhead and external visualisations, with interior detail and content coming from the story built by the reporter, programmer or desktop producer.  

                                     

Fig. 16. Images Google Earth

The  two major competing forces are Google Earth and Microsoft Bing. Both offer simulator quality photogrammetry video and in terms of legal and copyright clearance require at minimum an attribution, but recommend acquiring an image use license from the corporations themselves. Using satellite as well as images captured at street level, by a process called photogrammetry the algorithms are able to composite imaging, often as video and traversable in 3D, mapping the world, its oceans, rivers, lakes and mountains, in spectacular detail.

Whichever you use, as long as the content is actuality based, for overhead aerial viewing and story locating, these offer ‘simulation quality’ programming, not only for desktop but are increasingly cropping up in and featured in full-blown documentaries. A German company of artists and developers claim to have invented and passed on the ‘secret’ to successfully achieving this (see The Billion Dollar Code, Netflix). The outcome of a challenge to Google for the copyright to the algorithm for mapping the world was challenged and lost by the German group in the American courts (Balaga and Balaga, 2021). 

3D CAD SCENE RECONSTRUCTION 

This along with video photogrammetry exports I found were the most useful and important tools for any digital journalist and storyteller. An example of the CAD animations for the Ragamuffin App can be seen in Figure 15.  Searching and finding suitable 3D and 2D mesh objects is easier than one might imagine. The main learning challenge related to handling and manipulating the objects once imported.  An extremely powerful and liberating tool for media producers once mastered.

                                                 
Fig. 15. Ragamuffin App Rolling Stones Harbour Fight  (Weber, 2023. Personal collection)

You can access full texts of both academic papers on interactive film, behavioural analysis in critical situations and new digital methods in event reconstruction by making a donation to the website via PayPal.