What do vampires, The Matrix, and megalophobia have in common? They’re all discussed in this episode about the ways artists and scientists interpret and convey important messages. Whether it’s for building maps, or creating color palettes, guests Julia and Jake agree that physical experiences help to better understand our environments.
Transcript
Maoya: What do you do, and why do you find it interesting?
Julia: I'm Julia Oldham, and I'm a video artist working primarily with scientific concepts, but I also do a lot of deep dives into science fiction. A lot of my work is multi-channel video installation. By that, I mean, I'm projecting video onto multiple walls in a room and creating an immersive environment that you can enter. Something I feel really excited about is this possibility of creating a world that you can sort of go into and experience in a multi-sensory way. Something that I find really exciting about working with science is the way that science can offer a structure and system that allows me to construct unconventional and interesting narratives that I wouldn't be able to come up with without that outside fascinating, structural set of limitations.
As an artist, I lose a lot of control over the work as soon as it gets out into the world. I'm not really that interested in controlling people's interpretations or experiences with the work. It's very difficult for me to predict what work will be successful and what won't. What I really try to do is make work that's really authentic to my curiosity. My primary goal would be to have viewers be able to tap into that curiosity with me and to spark their curiosity, too. And to get them going down rabbit holes, whether that's a rabbit hole of eddy covariance or fire ecology or color theory.
Jake: I'm Jake Nelson, I call myself a data vampire. I take all of the data from other people that are producing it, and we take these eddy covariance data that people are measuring all over the world: carbon, water, energy fluxes, as well as meteorological data, so, temperature, humidity, incoming sunlight, and we match that with satellite data that give us kind of an indication of the greenness of the land surface or the surface temperature. We put all of that together from all the sites all over the world, and we make a statistical model. It's able to predict these things: carbon, water, and energy fluxes. Because the satellites are global, they're constantly going around taking images of the Earth, we can make a prediction anywhere we have a satellite image, of what we expect these fluxes to be.
A big one is what we call gross primary productivity, which indicates the photosynthesis of the CO2 taken up by plants, through photosynthesis. We can kind of translate what an individual tower or a network of individual towers making measurements corresponds to, say, how much photosynthesis is happening in the Amazon or what's happening in North America when there's a dry spell, these kinds of things. What's really fun and inspiring would be two things. One is that we're a big project, we're a big team: myself, Martin Jung, and Sophia Walther. It's not often in science you can really work day-to-day with a team, and along with that we get to actually interface with lots of different people all over the world that are taking these measurements and kind of understanding a bit of, not in-depth what every tower is doing, but a broad overview of it.
We really get to be integrators, which is a lot of fun. I was a plant physiologist, I was working in greenhouses and really small-scale and kind of had a curiosity of math and wanted to put those together. That fed me through the trajectory of my PhD, where I got linked with this group. Basically, this type of work is matching these flux towers with satellite data. It's a really old concept. I think originally, FLUXNET was about linking with satellite data. So, this had already been happening, and we were able to come in as a team and say, okay, we've been doing this for some years on the side. Let's really dedicate some time.
Maoya: Describe a little bit more about a project you're working on right now.
Julia: When I first started collaborating with Chris Still was last spring. The very first thing that I did was go to one of his research sites and climb a 180-foot tower, the US-Me4 AmeriFlux tower, close to Sisters. The goal of that was to see what the scientific process is like utilizing this tower and to learn about some of the tools that are installed on the tower. In the process of climbing this tower, which I did with Maoya, which was fantastic, I got to learn about the different eddy covariance tools. And, I also learned that there's a PhenoCam on many of these towers that’s taking a photograph of the landscape every half hour.
I got really excited about that and started thinking about possibilities for time lapse and for seeing time in landscapes in ways that is harder to do, usually. I started putting together time lapses using phenocam data from the forests near me in Oregon, initially, just as to sort of see how they changed, to see how the trees are growing, what they look like when they're growing. This eventually turned into a project where I began to translate each frame of those time lapses into a 10-color palette of stripes of color. So, abstracting the landscape into a color field, which was initially very much inspired by the fact that my collaborator, Chris, is looking at a data point called canopy greenness to learn about the forest. I thought that was such an exciting and rich point of overlap for me and him as an artist and a scientist to connect and talk and find these moments of connectivity.
My primary work that I'm making as a FLUXNET Artist in Residence is a three-channel project called September Orange, where I'm combining 24 phenocam sites throughout Cascadia, translating time lapses from those phenocams into color palettes that are paired with them, and then gridding all of those sites together so you can see the landscape and its color abstraction changing over time over the course of years. This project will ultimately be projected on three walls, and I have a special interest in finding moments of color unity among all of the sites. So, through a rather labor-intensive hand editing process, I'm scrubbing through timelines and finding moments of bright orange, and those ordinarily happen in September during our wildfire season.
Note: Sisters is a city in Deschutes County, Oregon, United States.
Maoya: Tell us a little more about what it felt like when you saw the first sequence of time lapses. What did you first notice?
Julia: Two phenocam sites that were really important for the genesis of this project were, firstly, AmeriFlux US-Me6, as a young ponderosa site. The camera has been running for about 13 years, and you're able to watch these baby ponderosas grow. I don't think I'd ever really seen a time lapse of a big pine tree growing. I was so charmed by what I learned is called "wicks"- the little new, bright green growths that emerge every year from the pine tree as it's growing, and they wiggle and are weirdly cute. It was so exciting to see that really animated part of the growing process and beautiful colors that emerge. This, like, chartreuse of the wicks - so wonderful.
I was primarily looking just at forests and then just out of curiosity, I took a look at an agricultural wheat field site in Washington State, and I generated a color palette to go with this time lapse. It's just a big green field. It looks sort of like the background of Windows - the Windows operating system. I thought, oh, well this will be kind of a gentle, beautiful change, and instead it was wildly dynamic. The growth of the wheat created these incredible textures - this bushiness. The field would get cut and you would see these interesting geometric forms from the cutting, and then suddenly there were bursts of the yellow that opened up in the field and spread and took over the entire field. And then, that yellow warmed into an orange. It was incredible. It made me realize that I needed to cast a wider net and look at a much wider range of landscapes to get a broader sense of what this place, this northwestern United States place, really is.
Maoya: Describe what it feels like to chug through data and to know when things are working, and know when things are right, when you're creating this data set.
Jake: It's looking at a lot of data in the same way that you're seeing these explosions of colors. When you say that, I can exactly imagine the time series. We look at a lot of these things we call "footprints" or "fingerprints". You see the hour on one axes, and the day of year on the other axes, and colors. And if you look at this, it looks like a fingerprint, and they're unique to every single ecosystem. Basically, I've looked so many of these, now, over the years that I can almost tell you which site is which based on these fingerprints.
It's just a lot of looking at the data, looking, do the time series look reasonable? Do the meteorological variables match what we're seeing from the fluxes? Does the model output the same kind of thing? It's a lot of data crunching, and then, once we get a map, we have to then look at that and say, okay, does this look like what we would expect? Making sure that what we're putting out is something that's useful for people and it's not going to lead people astray. So, it's maybe not as glamorous as it might seem, but it’s still fun.
Maoya: Just to give people like a sense of the scale, how many pixels are you dealing with, and what is the fraction of pixels that are measured and that are not measured?
Jake: So, we have about 350 sites. We work on an hourly time scale, so that's about, I think four million data points in the end that's going into the training data set. And that's the small part. The big part is what we run the model globally. So, then we have basically, each pixel is five kilometers by five kilometers, roughly, and every hour. Each time we make a prediction, it's about four terabytes of data that's coming out, and then typically we kind of slice and dice and aggregate that to a size that would be useful. Everyone's use case has a slightly different need, so maybe you just want a high-resolution map. Maybe you want time series. Maybe you want to look at the diurnal cycle.
Julia: I really loved what you just said about the fingerprint of a particular place and being able to know, kind of, what environment you're looking at through its fingerprint. It kind of reminded me of The Matrix, how you’re, sort of, able to make sense of the world just through looking at data. You know, as someone who's making a lot of sci-fi work frequently, I keep thinking that you're making, like, a ghost of a tower.
Jake: It is very much like that. We’re making artificial towers everywhere. Some are better than others, but you can't put a tower everywhere. It's similar that you're taking someone's data, and you're interpreting it. You're seeing what they've been measuring in a different way. Each of these sites has such a history. People have been measuring, like you said, 13 years. You have a phenocam running. You have a tower running. You have people going out and cleaning sensors. Meanwhile, that ecosystem is growing and changing and is alive. I'm just encoding it into bytes and computer code.
Julia: I hope that maybe something essential is happening, some sort of fundamental thing is being expressed, even if it is very simplified version of what I'm looking at.
Jake: I’m doing it in a way that feeds it into a computer, then it can understand it and translate it into a flux.
Julia: Something that's interesting for me, as I've been working on this project, is dealing with way huger amounts of data that I ever have before. As a video artist, hearing you talk about your 4 million data points - you're dealing with so much more data than I am. I'm just scraping phenocam sites and ending up with wildly huge piles of data that I have to figure out what to do with. I feel like I'm getting this little, tiny sense through looking at phenocam images of the vastness of data. It gives me a, like, megalophobia feeling.
Note: Megalophobia is an anxious sensation caused by immense structures, like skyscrapers, statues, or airplanes.
Jake: To make it worse, is it's only growing exponentially. The network started in the ‘90s. They had a release in 2015. From 2015 till now, so 10 years after, we’ve more than doubled the amount of data. They're planning another release, and it's going to double again in probably just the next few years. Figuring out ways to process it in meaningful ways becomes more and more important.
Maoya: We're getting more and more data, and as we're getting more and more data, we're finding more and more how unique different places are. Maybe each of you can kind of comment on that?
Julia: One of the sites that I've been looking at is the top of Mount Rainier. I am collaborating with someone who's doing some coding for me. He put together a code to help me find all of the reddest images. And at the top of Mount Rainier, we ran that code and I ended up with a thousand images that are a combination of a handful of forest fire days, and then some wildly crazy, beautiful sunsets. Also, some places where there's weird spider webs stuck to the lens, causing the image to completely blur and turn this weird red orange. It's really fascinating seeing these beautiful little particularities of each site caused by something like a spider visiting the lens. Those little tiny details are what help me to connect in this much more personal way to each site. Having this, like, mass of photographs, it can feel really overwhelming. But, finding a few ways of sorting through that gives you a strange, little bit of detail is really moving.
Jake: I interface a lot with AI (artificial intelligence) community. There's so much development with artificial intelligence and machine learning, and they have all these training data sets that are labeled "cat" or "dog" - these really curated datasets. The community wants to do something more practical: tackle climate change, we want to look at ecosystem fluxes. So, when we work with them, we give them this data and they get overwhelmed. They say, "This is noisy garbage. What are we supposed to do with this?"
You have to really explain and work with them one-on-one because it's exactly that. You look at something happening in the data, and sometimes you have to say, "Yeah, this is a particular species that behaves different." A lot of times, it's something that's happening strange here, and they say, "Yeah, the trees grew too much, and they had to raise the tower up", and it really changed what the tower was seeing, or a tree fell on the tower. So, being this interface, it's not just a really clean data set that you can sic your AI model on and it'll eat it up. You really have to understand what's going on, on the ground. A spider web in the sensor could cause an artefact in the data, and if you don't know that that's what happened, it can be hard to interpret. That’s a reality of making these measurements in these remote places with these complex instruments. That's exactly what we always have to explain, also, to our computer science friends.
Julia: I just wanted to share a story from a visit to Northern Arizona University. Recently, I spoke to Andrew Richardson's ecoinformatics class, and I was showing some of my work, which included a phenocam image where liquid had amassed on the lens and caused this, like, blurry blue image that I found wildly beautiful, and I was obsessed with. And, I started looking for blurry images in the phenocams. I wanted to find them all. At the end of my talk, one of the scientists raised her hand and said, "You know, when you first showed that blurry image, I thought, oh, man, that sucks - lost data." And I thought, "Yes, lost data! That's, like, going to be the title."
And then she said, "But then, I could see why you find it interesting, too." But I thought that was, like, such a wonderful, exciting representation of the way that, as artists and scientists, our goals and agendas wildly diverge. This moment of disappointment in lost data for her was a moment of excitement and abstraction for me.
Jake: If you’re interested in the plants, you see that, "Yeah we can't see the plant, it’s lost data." But if you talk to someone who's interested to dew, you just detected dew! You have a new sensor now.
Maoya: One of my projects was detecting fog in time lapse images, a long time ago.
Jake: How do you see the community? How do you see the people making these measurements, these crazy nerds that are super excited about this stuff?
Julia: Well, I had the opportunity to go to the AmeriFlux Conference last September. Something I was really struck by was the super-focus of data scientists, and how people can land on this, like, incredibly narrow, niche area of expertise, and know more about it than anyone in the whole world.
Maoya: Have you identified opportunities that scientists are, sort of, missing by being in these conferences with each other and not necessarily having the creative tools to go off-script of the science?
Julia: A, sort of, anecdote from this conference was that Dave Bowling came over and talked to me, he's at the University of Utah, about the time lapse work that I was doing with phenocam, and mentioned he was really interested in tracking and an analemma using phenocam. That's the figure eight shape that the sun makes if you take a picture of it at the exact same time every day over the course of a year. He said, "Andrew Richardson and I are thinking of setting up a camera to capture that. Would you be interested in, like, editing a video of it for us?" And I said, "Yes!" And then, as it turned out, there was already an analemma in the existing phenocam data.
I found one in a shadow moving through a greenhouse in Flagstaff. Andrew Richardson suggested I might just take a look at shadows in greenhouses to see, because then it captures, like, the opposite of analemma, of an inverted one. I spent about five days pouring through thousands and thousands of images. I just visually, sort of, found this site where the fish shadow stays within the frame for the entire year. I talked to Dave and Andrew, and said, "Hey, I think I found this." I downloaded a data set and then was able to animate on top of the analemma to draw the figure eight using my animation software.
So, we worked together to make this video capturing the analemma and put together our different sets of skills and expertise to make visible this really beautiful, weird phenomenon that happens in the world. There's been a few opportunities that have come up like that, where I've been able to use my software and skills to visualize something in a, sort of, simple way to make it clearer for a regular person to look at and see, and maybe, hopefully understand.
Maoya: Jake, can you relate to the difficulty or sometimes visualizing a result?
Jake: In all of these conferences, we usually have 5 to 12 minutes to explain what we're doing. We tried to develop video visual aids, very clean slides, that in one or two minutes convey to an audience [the] complex data pipelines that we have, and what we're doing. We tried quite a bit to really fine-tune and make this axis a little darker, let's use this different color scale. I really enjoy that aspect, giving a talk to someone, showing someone a figure. What about this distracts you? What about this do you get? Do you get the message I'm trying to convey?
Julia: I was interested in the visual abstract. I personally hadn't come across visual abstracts before. I was wondering how common that is, and whether that's something that you've ever been involved in making? My partner is a physicist, and I asked him whether he has seen visual abstracts before, and he said, not really, that that's not terribly common, at least in his field of physics. I found that really interesting and a really cool way of distilling information in this kind of designy way.
Jake: I actually made that. That came out of this, needing to convey the information quickly. That's typically how I start a presentation to really quickly jump in to give people an idea. A lot of journals these days will allow a video abstract or maybe even ask for one. It's just a skill that a lot of scientists don't have, or they don't have the time. It's kind of a shame because I think it's a really good way of disseminating the information and giving a brief overview that's not just text. What I found is when I do something like that, I typically use it a lot. So, I would always encourage scientists, now there's so many tools that can do this, to try and put these aspects in because they pay off in the long run.
Julia: I was really excited watching that and thinking about all of the people that something like a visual abstract can reach that might understand things more directly by looking and seeing.
Jake: For us, because we have these spatial aspects and this temporal aspect, showing a video of the fluxes, this gross primary productivity, you see an image of clouds moving through the frame and it gives you an idea of like, "Ah, okay, when the clouds come by there's less photosynthesis." <a href = "https://badgertalks.wisc.edu/speaker/paul-stoy/"Paul Stoy</a>, he's doing a lot with geostationary satellites where the satellite is taking in an image every few minutes or every hour, and really being able to show, "Okay, like, in this area of the US, crops haven’t been sewn yet, so you see nothing lighting up, and then suddenly, that explodes." It's information that is hard to convey in an image. You need, kind of, this dynamic thing that you see space and time, moving all at once.
Julia: Being able to go out into the field and directly observe physical implementation of experiments, going to a research site and watching Chris Still install a dendrometer, gives me a level of understanding of what he's doing that is very, very different than if I'm reading a paper or having someone explain to me what's happening. Getting a chance to observe that physical process, and the physical object, is huge.
These sorts of collaborations can create a really different understanding through experience, which I think is a somewhat-parallel to what we're talking about with something like a visual abstract. One of the things that I'm trying to do for the rest of my residency is film some of the devices, people's hands installing devices, getting some documentation of some of these actual objects to create a wider sense of what's going on.
Jake: I was teaching the course this spring. Each of the students, we basically gave them a tower to do a little exercise with. I really tried to convey, like, "This is a tower in Finland, and it's this kind of forest and it's been running for a long time", or, "This is a managed forest in Sweden, and they cut the trees", you know, "This is the person that runs the site, and they really like wheat beers in the evening" or something, you know, just like, some personal connection to all of these things and, and you remember more, and you understand the site more.
I was at a site near us, in Hainich Forest. It's one of the longest running sites in Europe, and I finally got to go out there this year. And suddenly you see, oh yeah, there was a patch of pines that was planted over here randomly, and they all died. If ever there's a flux coming from that direction, it's going to be very different than all the other surrounding beech forest. You wouldn't know that unless you had gone there and you saw, yeah, it's right there.
Maoya: Julia, do you have, like, a highlight of what you've learned from collaborating with flux scientists that has maybe influenced or changed your own approach?
Julia: The experience of climbing that tower as an introduction to this work that Chris and his colleagues are doing was such an amazing way to start. It allowed this shift in perspective, for me, that was both an intellectual shift where I learned about, "Oh, this tower does these things, and these scientists are utilizing this and this remote sensing tool". But also, it was a really extreme visceral shift because climbing that tower was terrifying. It was something that really pushed me into a different place in so many ways. I was doing something uncomfortable and frightening and exhilarating, but that’s at the heart of what my collaborator’s work is about. The tower as a focal point has been really, really important.
Another project that I'm working on is a four-channel, full-round, fully immersive video installation where I'm piecing together really slow-motion drone rises from the forest floor, usually just to the canopy. But sometimes, if I'm there at the research site with the scientists, I let the drone go all the way to the top of the tower, and I track people climbing it in slow motion. I'm trying to document that really unique, really fascinating element of research. It's super, super compelling.
Maoya: Explain how your work and this novelty of your approaches is most valuable for people inside your own community.
Julia: Kathleen Dean Moore is a writer and philosopher who does a lot of ecological writing. She was part of an art exhibition that I saw recently in Corvallis. She talked about how creative people might be a little bit like Perseus shopping Medusa's head off. And she went on to say, some things paralyze us. Medusa paralyzes us. Also, climate change might paralyze us, because it's terrifying, and it's depressing. Learning facts about climate change might make us feel hopeless and like we can't do anything. Kathleen Dean Moore suggested that, like Perseus, creative people might be able to hold up a mirror to a subject and make it more approachable.
Something that I hope that I can do, within my community and beyond it, is use my art to hold up a mirror to a subject like climate change and allow us to be unparalyzed and talk about it. I felt very moved by that metaphor of Kathleen's. It made me feel excited about approaching issues of environment and ecology and climate change as an artist.
Jake: By taking my data and integrating it into this regional aspect, it becomes a really, I hope, useful tool for both the public and politicians to understand how their ecosystems are using water, are they taking up carbon, are they losing carbon, what's happening to your local forest? And hopefully that convinces the funders of research how important these measurements are.
Maoya: Maybe you could tell us a little bit more of how those aspects of interdisciplinarity and reciprocity in flux science is key.
Jake: It's a voluntary network. Everybody's making their measurements for their own reason but also sharing the data for this broader ecological question of how the Earth works. When they do climate projections and they have these model ensembles that are projecting what's going to happen in the next 100 years, how much warming is going to happen, underneath that those models are evaluating against our products. They’re evaluating against the FLUXNET data, and I don't think that’s always seen. Just as the Earth's system is connected, so is Earth system science. We all have to borrow from each other and cooperate.
Qing: What role does wonder play in trying to address these overwhelming issues or, like, working at such a large scale?
Maoya: Why we started these artist residencies is I wanted to reignite a sense of wonder in the community.
Jake: I have to wear a lot of hats as a scientist. I have a lot of tasks and communication is one that I don't have any formal training on, and I don't have a lot of time to do it, but it's probably the most important. I was really excited about the podcast, I was really excited about the Artists in Residency, to just start telling some of these stories. If you're in North America or Europe, you probably have one of these towers in your backyard, and you don't even know it. And increasingly, in other parts of the world they're cropping up. So, it's conveying to people that this is actually their story, their ecosystems, their world. It’s really, really important, and cool.
Juila: As an artist working with issues of ecology and environment, I can come up with outlandish sci-fi solutions that aren't real, and that's okay for me to do. They might not be actual solutions to some of the problems that we're addressing, but they can maybe trigger conversations at least and spark curiosity and possibilities. There's a lot of imagination that becomes part of the process too, in my mind, as I'm moving through, like this forest, where there's all sorts of little devices on trees and bands around them, and some sort of device way up in a tree, and a giant box with zillions of wires dumping out of it. It makes something new happen for me. There's a lot of storytelling happening that is maybe filling in some of the blanks.