I made a device that mimics the sub-conscious of passive listening. The device records audio continuously. It then uploads the audio files to a remote location where those files remain in memory and are re-processed and combined to mimic the process of dreaming. Here the recordings are played back.
Prototype
I was able to get a working model in Processing using this code. I was able to leave the sketch running over night, recording in 1 hour intervals with no problems for 12 hours. This took up around 380MB at 44.1k 16 bit mono files. So I could get about a week’s worth of recordings with ~5.5GB.
One issue I had in the sketch was that using millis() was giving file sizes of varying degrees. I believe the problem was that in my IF statements and the fact that the operations would take longer than a millisecond, and therefore not get through the whole IF loop. So I jsut incremented a variable and had it reset to zero, giving me control over the interval in time as a function of the frameRate. I futzed around and got about an hour and twelve minute recording loop.
I took the files into Pro Tools when testing early on just to see how the transitions were, and it wasn’t all that bad, especially over a long interval. Here is a pic of the erratic bahvior of the millis() function set to record every 5 seconds. And a sample of that audio below.
Project Update
Here is a drawing/sketch of a fully functional “always-on” recorder. For now, I am just shooting to get the stand-alone portion working and recording locally (no cloud transfer) before I leave for Spring Break in two weeks.
With Danny’s help, I was able to get a sketch running in Processing that begins recording immediately. It saves the files as a pre-determined length with a manageable amount of dropped samples for the purposes of this device. I would like to make some way of having them overlap, but what could cause troubles when they’re being recombined. Danny and I also talked about a way to make a “mark” on each file so that you would know where and what to look for, like a button that appends a “check me” string onto the end of the file name. I could also knock the machine to make a loud sound which would be visible in an audio editor or could open a gate once the audio is analyzed.
On Sunday, I picked up a Raspberry Pi 3 Model B V1.2 from the ER, but once I got it home, I wasn’t able to power it on. I checked that my PSU was strong enough and reformatted the SD card and reloaded the OS, but nothing worked. I came back to ITP, got another Pi, this time it turned on, so I took it home and again, nothing. It would power on, but would not boot.
So today I came back and got a 3rd Pi, and Roland in the shop was able to help me get the OS up and running. First we tested that the Pi worked as another student (Mark from class) had a working Pi OS on an SD card. So once this new Pi booted up successfully, it was clear that my SD card was the problem. Roland directed me to go download Etcher which would reformat my SD card properly. I also downloaded the Stretch OS for Pi this time and not the NOOBS OS. I was able to load the OS ZIP file into Etcher and it formatted my SD card in a few minutes and… Pi booted up! I wish the Raspberry Pi website knew to direct folks to Etcher! I did not have time at school to test the Pi, but hopefully, during the next class, I can get the Processing sketch onto this Pi.
Research for Recorder
I checked out a Raspberry Pi B 3+ this week and tried to familiarize myself with it, but ended up not even being able to get OS loaded on to an SD card. There is a lengthy process where the SD card needs to be formatted in just such a way and it takes like 20-30 minutes each time and can fail right at the end :( I think that will be a viable option though at the B3+ has Wifi access, if I can get it to stream audio. Ideally the device would be able to save files straight to Google Drive.
I also investigated using iSpy which is security camera software, configurable for the home or office. It has the ability to take what appears to be web socket inputs and save those to the cloud. Here is a demo of that software (on my desktop Windows 7 machine at home).
Here is a list of web sites that I pulled up and saved in my research:
Getting Started with Raspberry Pi:
https://www.pubnub.com/blog/2015-05-27-internet-of-things-101-getting-started-w-raspberry-pi/Recording Audio on a Raspberry Pi:
https://medium.com/@PTcampbell/recording-audio-on-a-raspberry-pi-3a51ae351b98A USB Mic that could work with the Pi (I ordered one):
https://www.amazon.com/gp/product/B00IR8R7WQ/ref=oh_aui_detailpage_o00_s00?ie=UTF8&psc=1Record on Boot - Raspberry Pi:
https://www.raspberrypi.org/forums/viewtopic.php?t=61741
Then I got on a kick thinking I could record straight to Google Drive via the command line on the Pi, so that took me down a rabbit hole. I’d have to make a dev acct with Google to get a secure key so that I didn’t have to log in to Google Drive each time, which also means that to use my NYU account, I’d need special keys. Here’s some links on that:
How To Upload To Google Drive from the Command Line:
https://olivermarshall.net/how-to-upload-a-file-to-google-drive-from-the-command-line/Github repo on Google Drive CRL Client:
https://github.com/prasmussen/gdriveSetting up the proper authorization account for Google Drive (which i did):
https://developers.google.com/identity/protocols/OAuth2ServiceAccountAnd the Google Dev dashboard for my NYU acct:
https://console.developers.google.com/iam-admin/serviceaccounts?pli=1&project=always-on-recorder&supportedpurview=projectGoogle Cloud Platform (NYU)
https://console.cloud.google.com/navigation-error;errorUrl=%2Fhome%2Fdashboard%3Fauthuser
I also emailed back and forth with Tom Igoe on the matter. He pointed me in a few directions, like Chapter 10 of his book “Making Things Talk” as well as PhoneGap, as he suggested using a mobile phone to be the recorder. I though perhaps an old iPhone or iPod, since using one’s current mobile device would be tempting to be distracting in the moment of inspiration.
Dream, Vision, Goal, Plan
The Personal [Unconscious] Assistant
My conceptual goal is illustrate the the danger in using the conscious mind, with its attendant aims and sensibilities, to create devices that mimic and ultimately subvert unconscious processes, the needs of which are not well-understood.
Dream - To create a machine that records each and every experience of the conscious mind, just as our unconscious mind does automatically - from sight to sound, touch to taste and smell - enabling you to recreate an entire experience again in full contextual and sensory detail at will. You would have full conscious access, beyond the veil of time and forgetfulness, to the most significant and important events of your life from a wedding day, a career achievement, or one’s deepest childhood trauma, to something as mundane as a grocery list. This in many ways, could be considered a sort of Enlightenment Device - shining the light of consciousness into the darkest recesses of the mind.
Vision - Complete sensory recording - video and depth using two cameras, binaural sound, touch- and pressure-sensing fabric, a scent-o-graphic recorder, a tongue sensor and chemical sensor to analyze the content of the saliva - synced with master clock and live-streamed via the cellular network with the ability for users (or AI) to make markers and waypoints for later review. An AI algorithm would assess the content based on sentiment analysis based on the conscious inputs listed above.
In addition to these conscious inputs, users would also have the ability unconscious processes including exhaustive biometric data, including heart rate, steps taken including gate analysis, caloric intake and expenditure, pupil dilation, full dietary accounting, contextual elements like significant interactions and characters.
The program would also monitor external factors like bank account data, entertainment consumption and web browsing habits, text analysis via the visual input including SMS messaging, email, and social media accounts. All of this data along with the accordant family and ancestral histories in all of these categories as well as geo-political implications, allowing the user to thwart any impending threats to health or well-being of themselves or their loved ones.
Goal - To create a stand-alone machine that mimics the unconscious recording of aural sensory data. This device constantly records a live-stream of audio that is accessible at any point over a specified time-period of up to 24 hours - either remotely or on board via flash storage. This would be housed in a 6 x 6 inch box that plugs directly into the wall and runs on a Raspberry PI (withWifi if streaming).
Plan -
Feb 14 - get a Raspberry Pi and research recording software
March 7 - find a microphone and hook it up to the Pi
March 28 - (gone) find a way to loop or stream the audio either to the cloud or flash drive
April 1 - build a housing
April 18 - testing unit