DUUO: Fridge Inventory With 2 Old Android Smartphones - Smart Fridge

For a school project we made a 'fridge inventory' with two old Android smartphones to avoid waisting food.

We use the accelerometer of the smartphones and we also use speech recognition.

Once the fridge opens (with the smartphone attached to it), the speech recognition starts and you say to the fridge what you're putting in. For example: "apple". It will then be registered by the smartphone and will appear in a list on screen. We also keep a timestamp per product so you can see how long it's been in the fridge.

If you ate the apple, you're gonna throw away the core in the trashcan. Here you can find the other smartphone attached. Once the trashcan opens, the speech recognition starts and you say what you're throwing away. This information will be registered by the smartphone and sends it to the smartphone that's attached to the fridge. Here it will be removed from the inventory list.

So you can easily see on the list what products you have in your fridge and for how long they have been in there. When a product stays longer in the fridge, it get's another color (orange). When the product is expired (after for example 2 weeks), it turns red, so you know it is in fact expired and you can remove it from the fridge. But this way we hope we can avoid waisting food!

- 2 old Android Smartphones with a working accelerometer + chargers which can connect with the pc

- material to make a case for the phones.

Install here Processing for Android on your pc.

Important:

Put on 'Option for developers' on your smartphone + USB-debugging

Settings -> phone info -> build-number -> touch 7 times (you can find this on Google if you have another model)
Than setting: developer options -> check box USB-debugging

Make 2 new sketches in Processing and save these. For example under the names 'Fridge' and 'Trashcan'.

Put in each of these folders a 'data' map. In here you can put fonts and images for example. We used the font LemonMilklight (you can choose to use it of not, just put it on or off in the code).

First Install all libraries before we start coding.

• Ketai
• oscP5
• Timing Utilities (Lord of Galaxy)

// import

import android.view.View;
import android.content.Context;
import android.widget.Button;
import android.app.Activity;
///import android.speech.SpeechRecognizer;//in the "easy way that is useless
import android.view.Gravity;//for my layout
import android.graphics.Color;//for the color of the button
///import android.speech.RecognitionListener;///useless for the "easy way"
import android.speech.RecognizerIntent;

import android.content.Intent;
//import android.os.Vibrator;///let us take problems one after the other!!!!!
import android.widget.Toast;
import java.lang.Throwable;
import java.lang.Exception;
import java.lang.RuntimeException;
import android.content.ActivityNotFoundException;
import java.util.Locale;
import android.view.ViewGroup.LayoutParams;
import android.widget.FrameLayout;
import android.view.View.OnClickListener;
import android.content.Context;
import android.speech.SpeechRecognizer;

import ketai.sensors.*;
import lord_of_galaxy.timing_utils.*;

import oscP5.*;
import netP5.*;

//__________________________________

// declaratie van variablen
OscP5 oscP5;
NetAddress myRemoteLocation;
KetaiSensor sensor;
float accelerometerZ, accelerometerX, accelerometerY;
String afval;
Activity act;
Intent intent;
ArrayList result;
StringList inventory;
StringList vuilbak;
StringList timestamps;
StringList test;

color tekst = color(70,130,180); //blue
color BijnaVervallen = color(255,140,0); //orange
color Vervallen = color(255,0,0); //rood
color vergeetMijNiet = color(255,200,0); //yellow-ish
color achtergrond = color(220,230,255); //light blue

int h = hour(); 
int m = minute();
int s = second();

PFont myFont; //Only off you use your own font
PImage img; // Only off you use your own font

//settings
void settings(){
  fullScreen();
}

//setup
void setup(){
  act = this.getActivity();
  sensor = new KetaiSensor(this);
  sensor.start();
  
  oscP5 = new OscP5(this,12001);
  myRemoteLocation = new NetAddress("192.168.0.132",12001); 
// here you put the IP-adres of the smartphone which you wanna communicate with
  
  myFont = createFont("LemonMilklight.otf", 50); // here you can adjust the font
  textFont(myFont); // here you can put the font on or off
  orientation(PORTRAIT);
  textAlign(CENTER,CENTER);
  fill(tekst);
  
  inventory = new StringList();
  vuilbak = new StringList();
  timestamps = new StringList(); //time
  test = new StringList(); //time
  
  img = loadImage("test3.png"); // only if you use an image

}

//draw
void draw(){
  background(achtergrond);
  image(img, 0, -50); // only if you use an image
  
// if you wanna see the values of the accelerometer on screen, put this on
   //text("Accelerometer: \n" + 
   // "x: " + nfp(accelerometerX, 1, 3) + "\n" +
   // "y: " + nfp(accelerometerY, 1, 3) + "\n" +
   // "z: " + nfp(accelerometerZ, 1, 3), 0, 0, width, height);
  
    // tekenen lijst
  int j =75;
     
     for(int a = 0; a < inventory.size(); a++){
       text(inventory.get(a) + " - " + timestamps.get(a), width/2, 50 + (j*a));
       
     }
  
  //start speech 
  if (accelerometerZ < -1){
    startSpeech();
  }
  
 
 for(int p = 0; p < test.size(); p++){
   int banaan = int(test.get(p));

   
     if(banaan < millis() - 30000 ){
       //println(inventory.get(p) + "  " + " = " + "bijna vervallen"); 
       fill(BijnaVervallen);
       text(inventory.get(p) + " - " + timestamps.get(p), width/2, 50 + (j*p));
       fill(tekst);
   
     }
   
     if(banaan < millis() - 60000 ){
       //println(inventory.get(p) + "  " + " = " + "vervallen");    
       fill(Vervallen);
       text(inventory.get(p) + " - " + timestamps.get(p), width/2, 50 + (j*p));
       fill(tekst);
     }
 }
  
}

//onAccelerometerEvent
void onAccelerometerEvent(float z, float x, float y){
  accelerometerZ = z;
  accelerometerX = x;
  accelerometerY = y;
}

//currentTime
void currentTime(){
  text( h, 50,100);
  text( m, 100,100);
  text( s, 150,100);
}

//StringTester
String tester(){
  String blabla = str(millis());
  return blabla;
}

//wordTime
String wordTime(){
String hstr = str(hour()); 
String mstr = str(minute());
String sstr = str(second());

String time = hstr+ ":" + mstr + ":" + sstr;
return time;
}

//start Speech
private void startSpeech(){
   Intent intent = new Intent(RecognizerIntent.ACTION_RECOGNIZE_SPEECH);
   intent.putExtra(RecognizerIntent.EXTRA_LANGUAGE, Locale.getDefault());
   intent.putExtra(RecognizerIntent.EXTRA_LANGUAGE_MODEL,RecognizerIntent.LANGUAGE_MODEL_FREE_FORM);
   intent.putExtra(RecognizerIntent.EXTRA_PROMPT, "ik luister...");
   
   try {
          this.getActivity().startActivityForResult(intent, 666);///start Activity for result with some "code" (what you want) to "identify" your call
        } 
        catch (ActivityNotFoundException a) {
            Toast.makeText(this.getActivity().getApplicationContext(),"Spijtig, je telefoon heeft deze functie niet",
            Toast.LENGTH_SHORT).show();
        }
}

// onActivityResult
void onActivityResult(int requestCode, int resultCode, Intent data){
   switch (requestCode) {
            case 666: {
              if (resultCode == Activity.RESULT_OK && null != data) {
                  result = data.getStringArrayListExtra(RecognizerIntent.EXTRA_RESULTS);
                  background(255);
                  inventory.append(result.get(0));
                  timestamps.append(wordTime());
                  test.append(tester());
                  
              }
              break;
            }
   }
}

// message ontvangen
void oscEvent(OscMessage theOscMessage){
  afval = theOscMessage.get(0).stringValue();
  //println(afval);
  
  // message en stringlist vergelijken en verwijderen
  for(int i = 0; i < inventory.size(); i++){
   if((inventory.get(i)).equals(afval) == true){
     //println(i);
     inventory.remove(i);
     timestamps.remove(i);
     test.remove(i);
     break;
  }
}

}

// import

import android.view.View;
import android.content.Context;
import android.widget.Button;
import android.app.Activity;
///import android.speech.SpeechRecognizer;//in the "easy way that is useless
import android.view.Gravity;//for my layout
import android.graphics.Color;//for the color of the button
///import android.speech.RecognitionListener;///useless for the "easy way"
import android.speech.RecognizerIntent;

import android.content.Intent;
//import android.os.Vibrator;///let us take problems one after the other!!!!!
import android.widget.Toast;
import java.lang.Throwable;
import java.lang.Exception;
import java.lang.RuntimeException;
import android.content.ActivityNotFoundException;
import java.util.Locale;
import android.view.ViewGroup.LayoutParams;
import android.widget.FrameLayout;
import android.view.View.OnClickListener;
import android.content.Context;
import android.speech.SpeechRecognizer;

import ketai.sensors.*;

import oscP5.*;
import netP5.*;

// declaratie van variablen
OscP5 oscP5;
NetAddress myRemoteLocation;
KetaiSensor sensor;
float accelerometerZ, accelerometerX, accelerometerY;
String afval;
Activity act;
Intent intent;
ArrayList result;
OscMessage myMessage = new OscMessage("vuilbak");

PImage img; // alleen als je een afbeelding gebruikt

//settings
void settings(){
  fullScreen();
}

//setup
void setup(){
  act = this.getActivity();
  sensor = new KetaiSensor(this);
  sensor.start();
  
  oscP5 = new OscP5(this,12001);
  myRemoteLocation = new NetAddress("192.168.0.132",12001); 
//here you put the IP-adres of the smartphone which you wanna communicate with
 
  orientation(PORTRAIT);
  textAlign(CENTER,CENTER);
  textSize(36);
  
  img = loadImage("test4.png"); // only if you use an image
}

//draw
void draw(){
  background(78,93,75);
  image(img, 0, -50); // only if you use an image
  
// if you wanna see the values of the accelerometer on screen, put this on
  //text("Accelerometer: \n" + 
  //  "x: " + nfp(accelerometerX, 1, 3) + "\n" +
  //  "y: " + nfp(accelerometerY, 1, 3) + "\n" +
  //  "z: " + nfp(accelerometerZ, 1, 3), 0, 0, width, height);
  
  if (accelerometerZ < -1){<br>    startSpeech();
  }
  
    //message
  
  if(afval != null){
  oscP5.send(myMessage, myRemoteLocation);
  afval = null;
  myMessage.clear();
  }
  
}

//onAccelerometerEvent
void onAccelerometerEvent(float z, float x, float y){
  accelerometerZ = z;
  accelerometerX = x;
  accelerometerY = y;
}

//start Speech
private void startSpeech(){
   Intent intent = new Intent(RecognizerIntent.ACTION_RECOGNIZE_SPEECH);
   intent.putExtra(RecognizerIntent.EXTRA_LANGUAGE, Locale.getDefault());
   intent.putExtra(RecognizerIntent.EXTRA_LANGUAGE_MODEL,RecognizerIntent.LANGUAGE_MODEL_FREE_FORM);
   intent.putExtra(RecognizerIntent.EXTRA_PROMPT, "ik luister...");
   
   try {
          this.getActivity().startActivityForResult(intent, 666);///start Activity for result with some "code" (what you want) to "identify" your call
        } 
        catch (ActivityNotFoundException a) {
            Toast.makeText(this.getActivity().getApplicationContext(),"Spijtig, je telefoon heeft deze functie niet",
            Toast.LENGTH_SHORT).show();
        }
}

// onActivityResult
void onActivityResult(int requestCode, int resultCode, Intent data){
   switch (requestCode) {
            case 666: {
              if (resultCode == Activity.RESULT_OK && null != data) {
                  result = data.getStringArrayListExtra(RecognizerIntent.EXTRA_RESULTS);
                  afval = result.get(0);
        
                  background(255);
                  //println(afval);
                  myMessage.add(afval);
                 
              }
              break;
            }
   }
}

For the smartphones to be able to communicate with each other, they need to have each others IP-adres in the code.

Therefore you put the IP-adres of the phone attached to the trashcan in the 'Fridge' code.
And the IP-adres of the phone attached to the fridge in the 'Trashcan' code.

You can find your IP-adres under settings -> about the phone -> Status -> IP-adres

Remember: every time you reconnect with the internet, your IP-adres can change.

You have to ask for permission to be able to address certain functions on the smartphone.
Processing: Android -> Sketch Permissions.

The next boxes should be checked:

• BIND_VOICE_INTERACTION
• CAPTURE_AUDIO_OUTPUT
• INTERNET
• MODIFY_AUDIO_SETTINGS
• RECORD_AUDIO

Connect your smartphone to your pc with your charger. Select in processing: Android -> Devices -> 'your device'. After this you can click the play button in the upper left corner.

You install 'Fridge' on your smartphone that will be hanging on the fridge and 'Trashcan' on the other one.

Give the program some time to built and install. On some smartphones you have to click on 'install'.

In cas something goes wrong, check if the following bullets are correct.

• Are all the sketch permissions on?
• Are all the libraries installed correctly?
• Is your smartphone connected?
• Make sure your screen does not go in sleeping mode.
• Make sure everything inn the code is correct (for example: are the values of the accelerometer correct? / does the accelerometer work?)

// if you wanna see the values of the accelerometer on screen, put this on:

//text("Accelerometer: \n" + 
  //  "x: " + nfp(accelerometerX, 1, 3) + "\n" +
  //  "y: " + nfp(accelerometerY, 1, 3) + "\n" +
  //  "z: " + nfp(accelerometerZ, 1, 3), 0, 0, width, height);</p>

We've made a case for the smartphones and for a power bank with a laser cutter. We just tape it with double-sided tape to the fridge and to the door of the trashcan.

We used an MDF plate (600mm x 300mm) of 6mm.

You can of course always make your own designs!